MAKING CONNECTIONS

When you start talking to your customers about servers, they already know that the exciting stuff happens inside the box. It’s easy to upsell quad-core CPUs when the chips provide so much more processing power—even in slim 1U boxes—than single- or even dual-core designs. Of course, SMBs are going to perk up when you show them how much money they can save using enterprise SAS and low-cost nearline SATA storage in the same system rather than a bloated array of SCSI disks.

But servers are communications hubs above all else. They leverage all of the quad-core goodness and added serial storage technology to feed information elsewhere, whether that is across the world over the Internet or to a storage box in the same rack. Unfortunately, once you shift away from the latest and greatest headline-catching hardware and into the nitty-gritty of how those machines all talk to each other, non-technophiles start stealing looks at their watches.

The problem is not that communications hardware is boring or less important. In fact, you could argue that selling the right connectivity is tantamount to the success of your customer’s network. Where things seem to fall apart is when it comes to understanding the technology, especially for resellers who get thrown off by the enterprise vocabulary.

LAYING THE FOUNDATION
Let’s start with something easy: the foundation for almost all of the connectivity you’ll enable in a modern server, PCI Express. This newcomer to bus connectors is the serial successor to the PCI (and by extension, PCI-X) bus. Getting your head around the real benefits of PCI Express will help clarify the advantages of technologies like Fibre Channel, InfiniBand, and Ethernet.

While necessity has paved the way for wider and faster versions of the original PCI spec, including PCI-X and AGP, pushing it much farther proved cost-prohibitive a long time ago. The performance of a standard PCI bus is 133 MBps, derived from a 32-bit pathway running at 33MHz. And while you can multiply clock speed and bus width, doing so makes motherboard layout significantly more complex without yielding enough throughput to keep up with the 3 Gbps SATA ports, USB 2.0 hubs, and Gigabit Ethernet controllers typical of even today’s desktop boards.

One lane of PCI Express consists of an electronically simpler pair of signal wires transferring information at 2.5 Gbps in each direction. Because the technology is bidirectional, that raw bandwidth number doubles, yielding 500 MBps—almost four times what you’d see from PCI. Better still, PCI Express is highly scalable, making it easier for motherboard vendors to build boards with 1, 4, 8, or 16 lanes of PCI Express connectivity, pushing bandwidth up to 8 GBps without cluttering their designs. Those super-fast links are imperative to getting the most from add-on cards that might move data at 10 or 20 Gbps.

Multiplying Performance. The 82563EB Ethernet controllers (above)built into Intel’s SSR212MC2 storage server offer I/OAT, an advancement over the traditional performance-enhancing TOE.

FAMILIAR TERRITORY: ETHERNET
Now that you have a brief background on the high-speed bus technology that’s used to connect most motherboard components, it’s time to start dropping cards and chips onto PCI Express and work your way outside of the box.

Ethernet is the networking technology with which you’re probably most familiar. It’s been around since the ‘80s, delivering 10 Mbps of bandwidth, on through 100 Mbps and the 1 Gbps data rates more common today. The prevalence of Ethernet is overwhelming, and there’s a good chance that if you’re building a server like Intel’s new SSR212MC2 and sliding it into a rack, you’ll probably be connecting the machine to a Gigabit or even 10 Gigabit Ethernet switch, facilitating access to other machines on the LAN (local area network). In fact, the SSR212MC2 features two Gigabit Ethernet controllers built right on the motherboard.

Naturally, not all Ethernet cards or controllers are created equal. Traffic, performance, manageability, and room for expansion all help determine the solution you use. The good news is that the Gigabit Ethernet technology bundled with nearly every server board and a vast majority of mainstream desktop platforms is ample for general-purpose networking. For instance, the 82563EB Gigabit controllers you get on Intel’s SSR212MC2 platform connect to the 6301ESB I/O controller hub through a specialized bus and leverage support for the company’s I/O Acceleration Technology (I/OAT), which performs networking acceleration by having the network controller and chipset perform IP packet processing so the CPU doesn’t have to. Those two value-added chips are each able to push more than 100 MBps of information, letting you connect to two different networks or aggregate bandwidth to one.

The addition of Intel’s I/OAT bears special note, partly because it supersedes an integral component of most enterprise network adapters. It also doesn’t hurt that the feature is native to the 82563EB controllers. Perhaps you’re familiar with the term TCP Offload Engine, or TOE. The technology is prominently featured in many Ethernet cards as a means to shift the processing load tied to TCP/ IP from host CPU to network controller. You might be surprised how much compute horsepower it takes to process the TCP stack using only software at Gigabit speeds. A filled Gigabit pipeline can sack a 2 GHz CPU core. Adding TOE is a performance savior, especially as you push throughput beyond Gigabit.

Intel’s I/OAT takes that functionality a step further by operating on a platform level. It outperforms the traditional TOE approach through integration beyond the NIC. With processor, chipset, NIC, and operating system working together, I/OAT simultaneously delivers higher effective throughput and lower CPU utilization. The material gain from I/OAT comes into play when you deck a storage server like the SSR212MC2 out with twelve hard drives and install Microsoft’s Windows Unified Data Storage Server operating system, creating the fastest iSCSI SAN target possible.

Although a server board with dual Gigabit Ethernet connections gives you a lot of network bandwidth, there are plenty of situations where you’d want even more. The storage server example works here as well. If you’re connecting the SSR212MC2 to a D-Link DSN-3200, which has eight Gigabit Ethernet ports worth of connectivity, the two integrated connections aren’t enough to take D-Link’s SAN box to its limit. As your customer pushes up against the throughput limitations of one and then two Gigabit controllers, familiarize yourself with link aggregation and start adding more hardware. You can’t afford to let a bottleneck in connectivity kabosh thousands of dollars in surrounding server technology.

Packing it in. Three free PCI Express and two PCI-X slots let you install any combination of Ethernet, Fibre Channel, and InfiniBand controller boards in Intel’s 2U SSR212MC2 platform.

MULTIPLY AND CONQUER
Don’t think that you have to jump straight from Gigabit to 10 Gigabit Ethernet when the time comes to upgrade. Instead, link aggregation (formally IEEE 802.3ad) enables the teaming of ports to work in parallel, improving performance through aggregation and redundancy through fail-over. There aren’t any fancy hardware configuration tricks to worry about—each of Intel’s validated Ethernet NICs supports teaming.

“Port teaming on the SSR212MC2 is a matter of using the right software,” says Bryan Young, product marketing engineer at Intel. “One of the great things about using Ethernet is that almost all of the spec is supported, regardless of the card. It all comes down to software and how flexible it is. Linux, Windows Storage Server—even Windows XP supports link aggregation.”

So how do you go about loading up on Ethernet ports when it comes to expanding rackmount servers with very little room to grow? Check out Intel’s dual- and quad-port NICs, many of which leverage the performance advantages of PCI Express. The EXPI9402PT dual-port Gigabit card serves as the perfect example. It drops into a x4 slot, supports load balancing, and can be included in teams with up to eight Gigabit connections.

You’ll likely find Gigabit Ethernet to be the most cost-effective networking technology to deploy. The adapters are cheaper, the switching hardware is cheaper, and the copper cabling most SMBs use cost much less than the fiber optic cable common to enterprise networks. But once you’ve exhausted the throughput available to Gigabit, it’s time to look at 10 Gigabit and beyond.

A CONSOLIDATION PLAY. Within the confines of a rackmount server, there’s not much room to upgrade. Adding a card like Intel’s PRO/1000 PT Quad Port Gigabit adapter gives flexibility in confined quarters.

LOWER-LATENCY STORAGE
Ethernet is popular amongst SMBs because it’s inexpensive. As a result, iSCSI, which essentially boils down to running SCSI protocol over conventional Ethernet lines, is really taking off as an entry-level SAN technology. But when it comes to enterprise-networked storage, Fibre Channel remains the hard-hitting incumbent.

In comparing Fibre Channel connectivity to Ethernet, Intel’s Bryan Young says, “It’s a pure storage-based access media. The idea was that Fibre Channel wouldn’t be as cumbersome to set up and configure. It’s not routed like Ethernet. You buy a switch, plug in the auto-sensing ports—a NIC knows it’s a NIC and a switch knows it’s a switch—and go.”

Fibre Channel is billed as an enterprise storage technology, if only because of cost. Between the switches, controllers, and optical cables, your customer is looking at a much larger investment versus Ethernet. So what do you get after paying extra?

First, there’s the performance advantage. Dropping into a PCI Express interface opens the door to exceptional throughput, and you’ll find Fibre Channel cards from companies like Emulex running at 2 Gbps and 4 Gbps. While that’s already a substantial boost over Gigabit Ethernet, Fibre Channel is also more responsive, according to Young. “The packet overhead—or latency—is lower with Fibre Channel. That’s important in storage networking, which is why Fibre Channel is so popular.” Reliability also favors Fibre Channel technology. The error rate of optical cables is lower, and Fibre Channel establishes its connection before transmitting data packets. So when it comes to information getting from one place to the next, Fibre gets the nod.

The hard reality is that your SMB customers exploring network storage for the first time are probably in a better position to adopt an iSCSI over Ethernet infrastructure. Larger businesses already waist-deep in Fibre Channel are the ones who’ll persist down that path, but studies have shown that, especially for SMBs, the marginal benefits of Fibre Channel may not be practical compared to the substantial increase in infrastructure costs.

No Less Than 20 Gb. High-performance computing depends on screaming bandwidth. Mellanox’s InfiniBand cards can give your customers 20 Gbps of throughput in any available PCI Express x8 slot.

INFINIBAND: PERFECT FOR HPC
Like Fiber Channel, InfiniBand is a point-to-point, bidirectional, serial interconnect. Maybe you’ve heard of InfiniBand but have never had the opportunity to go hands-on with it. That’d be understandable since the technology has its roots in the high performance computing (HPC) world, linking nodes in some of the world’s fastest supercomputers. Increasingly, InfiniBand is being nudged down from those specialty applications toward more accessible Fortune 500s.

“The defining characteristic of InfiniBand is low latency,” says Dan Tuchler, senior director of product management at Mellanox. “The numbers are amazing—it’s one microsecond from one node’s application memory space to another memory space. That’s one microsecond past the OS, across the NIC, down the wire, and into the other memory space. We’re finding that a lot of applications are sensitive to that low latency. In particular, Oracle databases. When you whittle down the latency on messages passing between servers in a database, overall response time gets better.”

The InfiniBand card validated to work with Intel’s newest storage server is a 20 Gbps HBA from Mellanox that drops into the machine’s one PCI Express x8 slot. None of your other connectivity options even come close to that sort of speed. For the sake of comparison, Gigabit Ethernet gives you between 30 to 60 microseconds of latency, yielding less bandwidth and slower response times. Because Fibre Channel, which is also based on SCSI protocols, is only being used for storage, it isn’t even considered an InfiniBand competitor.

InfiniBand also gives you a better play on dependability. When an Ethernet network becomes congested, data buffers start filling up and dropping packets. TCP notices those dropped packets and retransmits, creating even more traffic and making the problem worse. An InfiniBand infrastructure employs credit-based flow control, which meters out transmissions to ensure that packets never get dropped.

According to Mellanox’s Tuchler, the clear market for InfiniBand remains clustering. But no longer is the enterprise-clustering concept out-of-bounds to the channel. Intel’s SSR212MC2 is InfiniBand-ready, as are some of Supermicro’s newest 1U server barebones systems. SMBs are only just getting their feet wet with HPC, which leaves plenty of room for VARs to innovate using InfiniBand as the glue linking compute nodes.

PUTTING IT ALL TOGETHER
For the reseller without any background in enterprise technologies like Fibre Channel or InfiniBand, understanding today’s connectivity protocols is the first step in taking them to market. Develop the vocabulary so you can start talking to customers about what they need. With this, you can then go back to your key vendor partners, discuss the sales opportunities, and gain the knowledge necessary to start having advisory and planning conversations with your clients.

High-end hookups aren’t just for large enterprises or HPC superusers anymore. Intel’s SSR212MC2 server can be configured for well under $5,000 and is fully tested with Fibre Channel cards from Emulex and InfiniBand HBAs from Mellanox. Additional PCI Express connectivity means you can also add quad-port Gigabit Ethernet cards to save space or a 10 Gb board for high-bandwidth applications. Don’t expect your customer to ask for network storage or an HPC cluster. Identify the need, know the technology, caution him about advantages and risks, and be a solution provider.