Friday, May 26, 2006
Data center networks often exclude Ethernet
For data center network managers, server interconnect technology falls into two distinct camps. For most, Ethernet, the world standard for networked computers, is how Windows, Linux, Unix and mainframe boxes are plugged in and accessed. But in the rarefied realm of high-performance data center clustering, technologies such as InfiniBand and some niche, proprietary interconnect technologies, such as Myricom’s Myrinet, still have a strong hold.
Over the past several years, InfiniBand switches have emerged as an alternative for some users. Makers such as Voltaire and Infinicon came to market with high-speed clustering switches that connect servers with specialized host bus adapters (HBA). These systems can provide as much as 30Gbps of throughput, with latency as low as the sub-200-nanosec range. (By comparison, latency in standard Ethernet gear is sometimes measured in microseconds -- one millionth of a second -- rather than nanoseconds, which are one-billionth of a second). This server-to-switch technology was so attractive that Cisco purchased InfiniBand switch start-up TopSpin a little more than a year ago for $250 million.
A need for speed, and more
“Ethernet is a good, versatile technology that can handle almost anything,” says Patrick Guay, vice president of marketing for Voltaire. “But Ethernet never had to address the levels of [traffic] efficiency and latency” required in clustered computer systems, storage networking and high-speed server interconnects, he adds.
“It’s not that there is no place for 10G Ethernet in data centers,” Guay says. “There is just a certain subset of customers who need more than what Ethernet and IP offer.”
This was the case at Mississippi State University’s Engineering Research Center (ERC), which runs several large Linux clusters used in engineering simulations for defense, medical and automotive industry research, among other areas. The ERC’s Maverick is a 384-processor Linux cluster connected by Voltaire InfiniBand products. Voltaire’s Intros 96-port InfiniBand switch is used to connect the diskless processor nodes, which access storage — and even operating system boot images — over the InfiniBand links.
This lets Roger Smith, network manager at the ERC, set up cluster configurations on the fly; however, many processors needed for a task can be called up quickly.
Smith says the communication between nodes in the cluster is "very chatty with very small messages" being passed back and forth. This requires extremely low latency among the nodes, so that no messages are missed, which could disrupt a job running on the cluster.
“Ethernet was just not ready for prime time, to get to the low-latency needs in some data centers” over the last few years, says Steve Garrison, marketing director for Force10 Networks, which makes high-speed Ethernet data center switches.
The latency even on older Ethernet networks is probably imperceptible for most LAN users — in the 1 to 50-millisec range. But in data centers, where CPUs may be sharing data in memory across different connected machines, the smallest hiccups can fail a process or botch data results.
“When you get into application-layer clustering, milliseconds of latency can have an impact on performance,” Garrison says. This forced many data center network designers to look beyond Ethernet for connectivity options.
“The good thing about InfiniBand is that it has gotten Ethernet off its butt and forced the Ethernet market to rethink itself and make itself better,” Garrison says.
Ethernet’s virtual twins
It’s become harder to tell standard Ethernet and high-speed interconnect technologies apart when comparing such metrics as data throughput, latency and fault tolerance, industry experts say. Several recent developments have led to this. One is the emergence of 10G Ethernet in server adapters; network interface card (NIC) makers such as Solarflare, Neterion and Chelsio have cards that can pump as much as 8G to 10Gbps of data into and out of a box with the latest PCI-X server bus technology.
Recent advancements in Ethernet switching components and chipsets have narrowed the gap between Ethernet and InfiniBand, with some LAN gear getting latency down to as little as 300 nanosec. Also, development of Ethernet-based Remote Direct Memory Access — most notably, the iWARP effort — has developed Ethernet gear that can bypass network stacks and bus hardware and push data directly into server memory. Improvements in basic Gigabit Ethernet and 10G chipsets have also brought latency down to the microsecond level — nearly matching InfiniBand and other proprietary HBA interconnect technologies.
The type of high-performance computing applications used at Mississippi State also has been the purview of specialized interconnects for a long time. One brand synonymous with clustering — at least in the high-performance computing world — is Myricom. The company’s proprietary fiber and copper interconnects have been in large supercomputers for years, connecting processors directly over the company’s own protocols. This allows for around 2 microsec of latency in node-to-node communications, and up to 20Gbps — more than 16 times faster than 10G Ethernet — of bandwidth. But even Myricom says Ethernet’s move in the high-performance data center is irrepressible.
“A great majority of even HPC applications are not sensitive to the differences in latency between Myrinet connections on one side and Ethernet on the other side,” says Charles Seitz, CEO of Myricom. The company is in the 10G Ethernet NIC market, having released Fiber-based adapters, which can run both 10G Ethernet and Myrinet protocols. This evolution was caused by customer demand for more interoperability with the Myricom gear.
“What do people mean when they say interoperability? They mean its interoperability with Ethernet,” Seitz says.
Universal acceptance
The widespread expertise in building and troubleshooting Ethernet networks, and universal interoperability makes it the better data center connectivity technology in the long run, others say.
“Ethernet does not require a certification,” says Douglas Gourlay, director of product management for Cisco’s Data Center Business Unit, which includes products such as the Fibre Channel MDS storage switch, InfiniBand (from the TopSpin acquisition), and Gigabit and 10G Ethernet products.
“With Fibre Channel, you had multiple vendors building multiple products, so the storage vendors took it upon themselves to create a certification standard for interoperability,” Gourlay says. “Now users won’t deploy products that are not certified by that specification.”
Gourlay adds that the industry is not at the point where InfiniBand technology, or other high-performance computing interconnect gear, has a common certification standard, or has proven interoperability among multivendor products.
“You can probably bet that you can’t build a network today with one of the narrowly focused high-performance computing [networking] technologies from multiple vendors.”
source:http://www.networkworld.com/news/2006/052506-data-center-ethernet.html
