Slow Linux router performance with VMware VMXNET3 adapter

I have a Linux guest under VMware ESXi 4.1 that runs OpenVPN and therefore acts as a gateway/router into the network. For weeks I was troubled with network slowness and weirdness. I knew OpenVPN wasn’t the problem because it’s fairly easy to configure and I’ve done dozens of similar deployments without any issues. Then I started reading some posts on the VMware forums about people complaining their Linux routers are basically unusable when using VMXNET 3.

I did happen to be using the VMXNET 3 adapter on this guest. It turns out this is a documented issue. The workaround suggests to set a module parameter option for “disable_lro=1″. I tried to do it using this guide but it didn’t seem to help me. For me, the solution was reverting back to using the more compatible E1000 adapter.

Read more about the issue in the vSphere 4.1 Release Notes:

Poor TCP performance can occur in traffic-forwarding virtual machines with LRO enabled

Some Linux modules cannot handle LRO-generated packets. As a result, having LRO enabled on a VMXNET 2 or VMXNET 3 device in a traffic forwarding virtual machine running a Linux guest operating system can cause poor TCP performance. LRO is enabled by default on these devices.

Workaround: In traffic-forwarding virtual machines running Linux guests, set the module load time parameter for the VMXNET 2 or VMXNET 3 Linux driver to include disable_lro=1.

My Solaris ZFS Home Server

I’m gonna start this new blog off right and dedicate my first post to my new Solaris based NAS/SAN build with ZFS filesystem. Since this is a home server my goals were the usual low cost/power/noise/heat/etc. For these reasons I chose a smaller mATX server motherboard and a cost effective Core i3-530 CPU combo. This CPU, when paired with the Intel 3420 Chipset on the motherboard, makes use of the ECC memory which ZFS requires.

The 6x 750GB RAID-Z (similar to RAID5) yields 3.3TB usable space. When attached to the LSI 1068e based Dell SAS6/iR adapter that I won on eBay for $25, they write at ~140MB/s and read at ~320MB/s. This build is fully capable of saturating gigabit ethernet, both upload and download.

OSOracle Solaris 11 Express
MotherboardSupermicro X8SIL-F
CPUIntel Core i3-530
RAMKingston 4GB (2 x 2GB) SDRAM ECC Unbuffered DDR3 1333
CaseAthena Power CA-SWH01BH8
Disk AdapterDell SAS 6/iR PCI-Express x8 SAS Controller
zpool (rpool)2x (mirror) 74GB Western Digital Raptor 10000 RPM
zpool (tank)6x (RAID-Z) 750GB Seagate Barracuda ES
$ pfexec mkfile 20g /tank/speedtest &
$ zpool iostat tank 30
 capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
tank        2.89T  1.17T     60      7  6.74M  97.4K
tank        2.90T  1.16T      1  1.28K  6.60K   138M
tank        2.90T  1.16T      0  1.33K     17   144M
tank        2.91T  1.16T      0  1.23K  10.8K   133M
tank        2.91T  1.15T      0  1.24K  2.70K   130M
$ pfexec dd if=/tank/speedtest of=/dev/null bs=16k count=1000000 &
$ zpool iostat tank 30
 capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
tank        2.91T  1.15T     60      7  6.74M   127K
tank        2.91T  1.15T  2.23K     25   285M   123K
tank        2.91T  1.15T  2.50K      0   320M      0
tank        2.91T  1.15T  2.30K     31   295M   167K
tank        2.91T  1.15T  2.44K      0   312M      0
tank        2.91T  1.15T  2.39K     30   306M   172K