Light At The End Of The Tunnel

We use the Exablaze Exanic X4 cards at work. The manual does explain what you have to do but doesn’t give you the CentOS/RHEL specifics. I’m assuming the driver is already installed.  The first thing to notice is that when the exanic.ko module is installed it creates the /dev/exanic0 device.  The permissions on this driver

crw-rw---- 1 root root 10, 58 Aug 27 21:46 /dev/exanic0

This is ok if your software runs as root.  We can change that by creating a /etc/udev/rules.d/exanic.rules

KERNEL=="exanic*", GROUP="exanic", MODE="0666"
KERNEL=="exasock", GROUP="exanic", MODE="0660"

In the above, you can see that I’ve changed the default MODE and the GROUP of the device so now when the module is installed the device is created with following permissions:-

crw-rw-rw- 1 root exanic 10, 58 Aug 27 21:46 /dev/exanic0

OK so now our software can read and write from the device.  To bridge the first two ports of the Exanic X4 you issue the command

exanic-config exanic0 bridging on

We usually implement the cards so that we log all the traffic out of the logging port, port 3.  This scenario is described on the Exablaze site. To do this we perform the following:-

exanic-config exanic0:0 mirror-rx on
exanic-config exanic0:0 mirror-tx on
exanic-config exanic0:1 mirror-rx on
exanic-config exanic0:1 mirror-tx on
exanic-config exanic0:2 mirror-rx on
exanic-config exanic0:2 mirror-tx on

This ensures that all the traffic we send and receive on a port is logged out of port 3.  This setting is stored between reboots. I usually prefer to put these entries in /etc/rc.local just to ensure that the card comes up in the mode I’m after.

For us our software bypasses the Linux Ethernet stack and therefore we can put each port on the Exanic X4 that we wish to use in a mode where the Linux kernel does not see the traffic on that port:-

exanic-config exanic0:0 bypass-only on

I again put this command into /etc/rc.local.

I have to admit I don’t like the use of /etc/rc.local but I haven’t looked at using if-local yet.

One of the nice features of the Exanic X4 cards is the fact that they can handle different speed and technology connections.  For instance, if I wanted to connect a 10G Fibre connection to a 1G copper Ethernet connection I can use port 0 of the card to handle the 10G connection, then install a 1G copper Ethernet SFP into port 1, then connect this port to the required 1G port. OK this sounds great, but port 1 does not automatically go into 1G mode, we need to set that in the /etc/sysconfig/network-scripts/ifcfg- file, as shown below:-

DEVICE="eth5"
BOOTPROTO="static"
HWADDR="64:3f:5f:01:17:a5"
NM_CONTROLLED="no"
ONBOOT="yes"
TYPE="Ethernet"
IPADDR=192.168.1.1
NETMASK=255.255.255.0
ETHTOOL_OPTS="autoneg off speed 1000 duplex full"

This now brings the port up at the correct speed.

One nice trick of the Exanic cards is that you can disable the port, and in so doing it will take down the link between it and the connecting device. This is great for diagnosing connection issues. However, if mirroring and/or bypass-ony mode are on then this does not work. You have to turn these features off before the port can be disabled.

The last feature I like is the ability to see the status of the installed SFP, i.e.

exanic-config exanic0:0 sfp status

Device exanic0 port 0 SFP status:
  Vendor: FINISAR CORP.    PN: FTLX8571D3BCL     rev: A
                           SN: ALG0C4Q          date: 111011
  Wavelength: 850 nm
  Nominal bit rate: 10300 Mbps
  Rx power: -1.9 dBm (0.65 mW)
  Tx power: -2.1 dBm (0.62 mW)
  Temperature: 49.8 C

This is another feature useful diagnosing connectivity issues.

Let me know if you have a better solution to /etc/rc.local.