The IEEE-ISTO (International Standards and Technology Organization) held its first conference on product certification and conformance at their IEEE headquarters in New Jersey. The goal of the IEEE Conformity Assessment Program (ICAP) is to provide support to other IEEE standards groups, test labs, and industry groups in developing conformance tests. It's a first step on a long road for the ICAP.The IEEE isn't getting into the certification business, rather ICAP wants to facilitate the development of conformance certification using existing groups and labs and this workshop got that conversation going. ICAP is a marked change for the IEEE where their involvement with standards, in the words of Rudi Shcubert Director of ICAP, ended after the standards were developed and published. It sounds good in theory, but after a half day of presentations, from NIST, the Wi-Fi Alliance, the Ethernet Alliance, and OmniAir, it's clear that developing a conformance program is an ambitious effort that will take time to grow. I hope they are successful.

Gordon Gillerman from NIST offered up a definition of conformance quoting ISO/IEC 17000 as the "demonstration that specified requirements relating to a product, process, system, person, or body are fulfilled." The definition much sums up what we, as consumers of products, want. But in the case of IT standards, we need to specify that products which implement a standard are also interoperable. It became clear on the first day of the conference that conformance and interoperation have very different goals. The assumption that conformance to a standard means that products will interoperate is a bad one. Interoperation according to standards is what we really want and we can only ensure that through interoperation testing and validation. Often times, the people who work on standards documents are not the same people that are interpreting standards documents during product development. That is one of many gaps that ICAP wants to address on the way to conformance certification-aligning standards development with product development.

Gillerman noted three types of certification.

1. Suppliers declaration is when a vendor claims to adhere to a standard. This method is used when the risk of noncompliance is low, ie not life threatening, and there are penalties and methods to remove non-compliant products from the market. For example, if you bought ISO 400 speed film, but found it was ISO1600, you'd probably not buy that film any longer.
2. Inspection is when the critical characteristics can be measured or examined. For example, an electrical inspector has to examine and approve electrical work done in your home after the wiring is complete but before power is applied.
3. Certification is when the risks of non-conformity are high and includes evaluation, attestation by a 3rd party, and surveillance or follow-up testing. For example, if you put 10w30 motor oil in your car, but it was really some other weight, you could damage your car. The American Petroleum Institute manages the certification for motor oil.

It's good to keep this taxonomy in mind when thinking about conformance certification because not all efforts require the same sort of certification, much of which depends on what Gillerman calls the demand driver-more often than not, money-which flows through many sources. Consumers expect their high definition equipment to work together. Enterprises expect products to interoperate and are a demand driver when they tell vendors to support standards, a point I made in Standards Matter: The Battle For Interoperability Goes On. Manufacturers demand suppliers adhere to quality standards, and so on.

One of the better known interoperability certification groups is the Wi-Fi Alliance. As Greg Ennis from the Wi-Fi Alliance pointed out, early implementations of 802.11b access points and NICs, though standards conformant, didn't interoperate. You had to purchase both components from the same vendor to ensure it would all work. However, as Wi-Fi became more widespread and embedded wireless showed up in many devices, there was no way that situation was untenable. Individuals, enterprises, and retail outlets where the demand drivers telling the Wi-Fi vendors to interoperate or else. The Wi-Fi Alliance brought order to that chaos as more vendors were certified interoperable. When the WEP debacle broke, the Wi-Fi Alliance stepped in with WPA while the IEEE working group addressed the weaknesses in WEP. Then the Wi-Fi Alliance aligned WPA2 certification with 802.11i. Similarly, while 802.11n got bogged down in standards work, the Wi-Fi Alliance's expanding role created certification for a draft standard of 802.11n while the IEEE work group is completing their work.

Brad Booth from the Ethernet Alliance says the group is trying to develop something similar. The Ethernet Alliances is a recurring group that forms up around an Ethernet standard like fast or gigabit Ethernet. The group works on interoperability testing among vendors, in his words behind the scenes, that ensures basic interoperability between switch and NIC vendors. There are few interoperability problems at layer 1-2 between switches and NICs today, but there has never been a certification program. With the advent of 10Gb Ethernet and Data Center Bridging-the suite of protocols that define lossless Ethernet QoS and priority forwarding which are required for SAN protocols-simple interconnectivity gets more complex. If one of the goals of DCB is to shed the manacles of the qualified product lists that exist in storage networking today, enterprises will demand proof that products interoperate seamlessly and well. Booth thinks that functional certified interoperation is within reach, but Booth shied away from the very thorny issue of performance validation.

Performance claims by vendors, as we all know, are often not reliable. The actual parameters of the tests are usually configured to show best performance of the product and hide problems. Such shenanigan makes it hard for enterprises to find out how a switch will perform without putting it in-line. Even the product literature is misleading. Many vendors of 24 or 48 port 10Gb switches claim wire speed, 96GB (48Gb full duplex), performance, but fail to mention that is per port and not through the switch. Selective disclosure is annoying to IT buyers that are familiar with the slow ramp-up of devices from oversubscribed products to wireline non-blocking products. First comes connectivity, then port density, then non-blocking full capacity through the switch. IT buyers don't get annoyed if vendors tell them that the capacity is less than full line rate on all ports. They do get annoyed with that little detail is left out.

In addition to protocol conformance testing, performance validation is necessary-the ability to test a vendors claim that their product performs as advertised in a fair and uniform manner. I asked Booth about performance testing after his talk and he had two responses.

1. The first was that Ethernet Alliance members, remember it's an industry group, wouldn't want performance validation testing for fear of not being fastest. That's ridiculous. Enterprises don't buy switches based solely on performance. If performance is an issue, hiding the true performance won't win the vendor any points.
2. The other reason Booth gave is the difficulty in making up the test scenarios that would be meaningful.

Coming up with meaningful performance tests is difficult, but useful models could be developed. Using IETF RFC 2889 "Benchmarking Methodology for LAN Switching Devices" and RFC 2544 "Benchmarking Methodology for Network Interconnect Devices" as models, test specifications could be written and finalized. Booth had a nice chart of a certification timeline where he shows that certification is in demand right after a standard is complete, but as the market matures, conformance testing demand diminishes because in Gillerman's terms, non-conforming products won't survive in a mature market. I'd posit that demand for performance validation follows the same curve right now for 10Gb. For the foreseeable future, enterprises want to see performance validation.

It's good that the IEEE-ITSO is having this conference and bringing together organizations in various stages in conformance programs. The group plans more meetings with other speakers in a hope to jump start conformance testing within the IEEE. I'd like to see it succeed and I'd like you, dear reader, to tell your vendors that conformance and performance validation are necessary.