Line Rate

"There are two kinds of people in this world..." I hear this from my father-in-law on a weekly basis. It's usually followed by some quip like "the quick and the hungry".

Twinax cabling (aka direct-attach SFP+) is a cost-effective way to interconnect switches, servers, and storage at 10GbE. Current implementations are limited between 5M and 10M. The economics are quite compelling when compared with discrete SFP+ optics. Well, there are 2 kinds of 10GbE twinax cables in this world: active and passive. (My father-in-law is beaming right now). Unfortunately, not all NICs or switches support both standards.

I was helping a client to migrate their VMware ESX 4.x servers to redundant 10GbE interfaces. They had Dell servers with OEM Intel NICs. They had done some initial testing with Dell's 24-port 10GbE switch, but ended up selecting Brocade's VDX6720 Ethernet Fabric network for redundancy and growth prospects. Once we had everything cabled and powered-up, there were connectivity issues. No link-up and the VMware host gave us purple-screens on shutdown.

A little poking in the /var/log/vmkernel logfile showed this:

"vmnic8: ixgbe_sfp_config_module_task: failed to load because an unsupported SFP+ module type was detected."

Well, come to find out that Intel's 10 Gigabit AF DA Dual Port Server Adapter isn't compatible with the active twinax cables that came with the Brocade VDX switches. The VDX switches don't support the passive cable that came with the Intel 85298-based 10GbE NICs.

This customer also had a handful of the newer Intel Ethernet Server Adapter X520 Series sporting the newer Intel 85299 ASIC. We still had problems getting things working. A little more digging shows that active twinax support for the x520 series was a semi-recent addition and according to Intel, requires driver version 15.3 or later. We spun up a copy of vSphere 5.0 hoping that it included a newer version of ixgbe. It sure does and we now have 10GbE over active twinax between the Brocade VDX6720s and the Intel x520 NICs.

Moral of the story: get your active/passive cables straight. Make sure your NIC & switch support the same standard. Make sure your drivers are recent enough (at least for the x520) to support the correct types of cable.

I work in the networking industry and have a 19" rack in my home office. I've gotten tired of having to string serial cables across the room to a specific device in order to get a basic config on it.

I picked up Web Power Switch from digital-loggers.com to handle Ethernet-based power control. I also picked up an Avocent/Cyclades ACS32 serial console unit on eBay ($150-ish). They both work great, although I couldn't get console access to the Foundry / Brocade gear working.

I pulled out the user guides and through some trial and error finally figured it out. You'll need some DB9 Female to RJ45 Modular Adapters, and an "insertion and extraction" tool if your dongles have the pins already inserted.

You can use a standard CAT5 Ethernet cable for the RJ45-side of things. On the DB9 dongle, you'll want to map the following colors to these pins:

DB9 Pin - Color
1 - White (not needed)
2 - Yellow
3 - Black
4 - Orange (not needed)
5 - Red
6 - Brown (not needed)
7 - Blue
8 - Green
9 - no connection

The "not needed" lines came from the Foundry manuals. It looks like their serial console ports only need TX/RX CTS/RTS and GND. The other pins needed to go somewhere, and since I didn't want to cut them off, I just plugged them in the other ports randomly.

Now I can sit at my desk and power on a network device, remote console into it, give it a config, do my testing, and power it off again, all without having to leave my desk.

Hope that helps someone out there. Good luck!