VMware Networking Speed Issue

Keywords that you may find useful relating to this issue: super slow network, failure to connect, transmit, vmware, virtual machines, network adapter, network card, E1000, vmxnet, vmxnet2, vmxnet3, disable TSO, disable GSO, segmentation offloading

Banging your head against the wall with strange network speed issues that seem to come out of nowhere? Perhaps you can't connect to certain machines from other virtual machines?

Cause:

There is a bug related VMware network adapters or their drivers related to "segmentation offload" (TSO and GSO). I have heard one mention that the checksum feature may also have problems.

Solution:

Turning off TSO/ESO on the guest OSes is the fool-proof solution. You could try installing a newer OS or try playing with the adapter type (eg: E1000, VMxnet2) but that seems to work only some of the time.

What is TSO:

TSO and GSO (standing for tcp segmentation offload and generic segmentation offload) means that instead of the CPU breaking up packets, the network card is expected to do it.

Consequences of turning it off:

Having it on improves performance by reducing CPU load. Many reports say that they don't experience CPU speed issues until they reach ridiculous amounts of sustained data (>50Mbps). Other than that, it will speed up performance to disable it according to many, and certainly will if you are grinding to a halt due to this TSO bug.

Test It Out:

On Unix, you can usually check if TSO and GSO are on by running

and you can disable them it by running

If it says "operation not supported", you may want to still try setting only the TSO or only the GSO.

If ethtool isn't installed, do the following:

On Windows, go to 

You may also want to look for things like "IPv4 Giant TSO Offload", I turn them off to be safe but I haven't actually confirmed if they matter or not. Look over and check for any other Tso related settings too.

Do It On Reboot For Unix:

There are many possible ways of doing this. You may think to try adding a line to /etc/network/interfaces, changing /etc/networking/if-up.d/ethtool, or even changing ETHTOOL_OPTS= that is set elsewhere and passed into if.up.d/ethtool. Those ways allow may seem to work after an ifup but they may fail during bootup because there may or may not be other steps (like with vmware-tools) in the bootup process that change the TSO back on.

To find the offending places, you may run:

Since we prefer to see the settings easily in the same place you would check your IP and other settings, we went with this 2 step approach:

1. Comment out or change any lines that the above grep shows to change tso to on. Often this may be /etc/init.d/vmware-tools.
2. In case the above is not affecting all the nics or vmware-tools is not installed or otherwise doesn't have any lines for TSO on and the default is to have it on, we still add this to /etc/network/interfaces:

It also serves as a great reminder if you go there to check network settings and notice it missing.

If You Found This Pleasant Solutions Blog Entry Useful

As always, if you found this article useful and you have a blog of your own, a quick mention would be always be appreciated. If you saw another place where people were asking for this information and did not have an answer yet, helping them out would also be good.

References:

 Vmware Specific:

• Mentions that the host itself can "Detect Tx Hang" for any physical hardware issues on the host (disabled with UseHwTSO setting): http://www.vmware.com/support/vsphere4/doc/vsp_esxi40_u1_rel_notes.html
• VMware giving instructions on how to turn TSO on using registry for some Windows systems: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1465
• Mentions that, as of v4.1, VMware no longer has the ability to disable TSO in the VMware config file without having the guest OS do it: http://communities.vmware.com/thread/281839;jsessionid=6E85AD562BE0A5DE1AB8460E05DA29FF?tstart=50

Not-Vmware Specific:

• Snapshots of where to disable TSO in Windows: http://seer.entsupport.symantec.com/docs/294308.htm
• Shows that checksum offloading can sometimes cause problems to: http://www.novell.com/support/search.do?cmd=displayKC&docType=kc&externalId=7005304&sliceId=1&docTypeID=DT_TID_1_1

Unix Related:

• Turning off TSO even after a reboot: http://ubuntuforums.org/archive/index.php/t-247317.html and http://www.debian-administration.org/articles/28
• /etc/network/if-up.d/ethtools gets its settings from http://www.redhat.com/archives/broadcom-list/2003-December/msg00002.html