Network Bonding in Gentoo

I’ve been trying to find a good use for the second network adapter in my fileserver, and what better than to increase the throughput to the network. Using something called bonding, its possible to combine two physical network adapters into one logical bonded adapter. Using a bonded adapter provides several advantages such as fallover should one network fail, but more interesting for me, increased throughput.  There are several different methods that one can use to bond the adapters together.  The best way to double the throughput of the bonded adapters is to use the IEEE 802.3ad protocol, however this requires support on the switch.  802.3ad support is generally limited to managed switches and is not supported on my unmanaged Netgear JGS516. The next best thing is to use a special mode of the bonding driver in Linux, called balance-alb.  The other modes supported by the kernel bonding driver allow things such as fallover.  I’ve been told that the following method does not work with all network adapters. Specifically, the driver needs to be able to change the MAC address on-the-fly, which is not supported in all drivers. I can say that it works with the r8169 (Realtek) and the Intel drivers (e100, e1000, etc.). This post is written with Gentoo in mind but should apply to other distributions.

First thing you need to do is enable bonding in the kernel.  Be sure to compile it as a module since we will need to pass arguments to the module when it is loaded:

Device Drivers ---> Network Device Support
<M>   Bonding driver support

Now set the module to be loaded at boot:

# nano /etc/modules.autoload.d/kernel-2.6
bonding mode=balance-alb miimon=100

The two options above are important. The balance-alb (adaptive load balancing) specifies what bonding mode to use.  There are several different options available here, but balance-alb provides the functionality I want since my switch does not support 802.3ad bonding.  The second option, miimon=100, tells the module to use mii-tool to check if the network adapters are up every 100ms.  This provides the fallover should one adapter fail.

Now emerge ifenslave, a userland tool to bond the interfaces:

# emerge -av ifenslave

Next, we need to configure the network interfaces:

# nano /etc/conf.d/net
slaves_bond0="eth0 eth1"
config_bond0=( "dhcp" )
config_eth0=( "null" )
config_eth1=( "null" )

This bonds eth0 and eth1 to form the bond0 interface.  You can bond more than two adapters as necessary.  The bond0 interface is configured to use DHCP to automatically obtain an IP address.  The config_eth0=( “null” ) lines prevent the individual adapters from getting an IP address since we only want bond0 to get an IP.

Now lets create the startup script, have it start automatically on boot, and remove the individual adapters from starting at boot:

# ln -sf /etc/init.d/net.lo /etc/init.d/net.bond0
# rc-update add net.bond0 default
# rc-update del net.eth0
# rc-update del net.eth1

That should be it, reboot, and you’ll now have bonded adapters.  I used a program called netio to measure my network speeds.  Prior to bonding, I could read at about 60-80MB/sec from the fileserver over a gigabit network.  With bonded adapters, that increased to about 120-150MB/sec, quite an improvement!  Write speeds saw a significant improvement as well.  Note that many hard drives will not be able to read/write at the same speed as the network adapter.  So if your adapter can push out 150MB/sec, it doesn’t help you if your hard drive can only read at 60MB/sec.  Internally, the RAID 5 array in my fileserver can read at up to 250MB/sec using hdparm, so the array can more that keep up with the bonded adapters.  If you have an older computer with dual 10/100 adapters, bonding can give you a nice boost to throughput that most hard drives will still be able to keep up with.

References and additional information:

Boost Reliability with Ethernet Bonding and Linux

Tips and Tuning for Ethernet Bonding With Linux

Gentoo Linux Documentation — Modular Networking

Expand JFS Filesystem

I recently bought 2 new 500GB hard drives and am in the process of migrating my RAID array to double its size to 2TB. The current 1TB over 3 drives was 100% full, so it was something that had to be done. The expansion took about 26 hours on my 3ware 9650SE-8LPML controller, which is quite good from what I see of other peoples experiences. Of course, all important data was backed up before the RAID expansion.

The primary issue I had after the expansion was the proper partitioning of the free space. When the RAID expansion was complete, I now had 1TB of used space using the JFS filesystem that was the original partition, and 1TB of unused, unpartitioned free space. Attempts to use Gparted to expand the partition failed due to a known bug that prevents partitions over 1TB. I tried to use Parted, the CLT tool that Gparted is based on, to expand the filesystem but was unable to do so. This seemingly left me with two options, either use multiple smaller partitions, or to reformat and restore the data from the backups. I decided that smaller partitions would be a lot of extra work in terms of keeping tabs on the free space on each partition (and may require more frequent partition modifications) which is something that should not be necessary. Restoring from backups was not welcome due to the time involved. Deciding that those options were not acceptable, I was determined to find the solution I wanted, having a 2TB partition without going through a backup restore process. If anything, it would be worth doing simply on the principle of the matter: that having large partitions on a modern filesystem that supports volumes up to 32000TB should not be a problem. Several Google marathons and man page studies allowed me to successfully perform the operation that I wanted.

My main problem was a lack of understanding of the way that fdisk actually works. With fdisk, you can delete and recreate a partition without actually destroying the filesystem that lives on the partition. I am not sure exactly what the limitations are, but it seems that as long as you do not change the starting point of the partition, the filesystem will remain there. In my case, the drive in question (/dev/sda) had a 1TB partition starting at the beginning of the disk (/dev/sda1) with 1TB of unpartitioned free space residing after the JFS partition (created by the RAID expansion). I deleted the JFS partition (/dev/sda1), and recreated the partition, but this time using the full 2TB of space. In the tests that I did, the free space MUST come after the partition. I wrote the partition table, and them mounted the drive. All the data was still there, but the JFS partition was only showing up as 1TB with the command dh -h. This was rather concerning until I realized the state that the drive was in. fdisk only edits the partition table, not the actual filesystem. When it writes out the partition table, it is just redefining where the partition begins and ends and does not touch the filesystem. Thus, although I had expanded the partition, I did not expand the filesystem. Doing some more searching, I found this article that gave me the last piece of information I needed. The command mount -o remount,resize /mount/point tells the JFS filesystem to expand to fill the entire space of the partition. The command is unique to JFS and will not work on other filesystems because the options passed are used by the JFS kernel driver. When you issue the mount command, note that the partition must be mounted.

Perhaps most frustrating is the lack of data on the JFS filesystem. It does not seem to be very popular in the Linux community, most of whom use ext2/ext3. Those that do use an alternative filesystem tend to use ReiserFS with a small number using XFS. In the limited, unscientific testing I have done, JFS performed very well performance wise and used far lower CPU than ext3 or ReiserFS. JFS has worked out well for me so far and, despite my initial difficulty in expanding the filesystem, is the filesystem I will use in the future.