Line Rate

"There are two kinds of people in this world..." I hear this from my father-in-law on a weekly basis. It's usually followed by some quip like "the quick and the hungry".

Twinax cabling (aka direct-attach SFP+) is a cost-effective way to interconnect switches, servers, and storage at 10GbE. Current implementations are limited between 5M and 10M. The economics are quite compelling when compared with discrete SFP+ optics. Well, there are 2 kinds of 10GbE twinax cables in this world: active and passive. (My father-in-law is beaming right now). Unfortunately, not all NICs or switches support both standards.

I was helping a client to migrate their VMware ESX 4.x servers to redundant 10GbE interfaces. They had Dell servers with OEM Intel NICs. They had done some initial testing with Dell's 24-port 10GbE switch, but ended up selecting Brocade's VDX6720 Ethernet Fabric network for redundancy and growth prospects. Once we had everything cabled and powered-up, there were connectivity issues. No link-up and the VMware host gave us purple-screens on shutdown.

A little poking in the /var/log/vmkernel logfile showed this:

"vmnic8: ixgbe_sfp_config_module_task: failed to load because an unsupported SFP+ module type was detected."

Well, come to find out that Intel's 10 Gigabit AF DA Dual Port Server Adapter isn't compatible with the active twinax cables that came with the Brocade VDX switches. The VDX switches don't support the passive cable that came with the Intel 85298-based 10GbE NICs.

This customer also had a handful of the newer Intel Ethernet Server Adapter X520 Series sporting the newer Intel 85299 ASIC. We still had problems getting things working. A little more digging shows that active twinax support for the x520 series was a semi-recent addition and according to Intel, requires driver version 15.3 or later. We spun up a copy of vSphere 5.0 hoping that it included a newer version of ixgbe. It sure does and we now have 10GbE over active twinax between the Brocade VDX6720s and the Intel x520 NICs.

Moral of the story: get your active/passive cables straight. Make sure your NIC & switch support the same standard. Make sure your drivers are recent enough (at least for the x520) to support the correct types of cable.