VMware Networking Speed Issue

Keywords that you may find useful relating to this issue: super slow network, failure to connect, transmit, vmware, virtual machines, network adapter, network card, E1000, vmxnet, vmxnet2, vmxnet3, disable TSO, disable GSO, segmentation offloading

Banging your head against the wall with strange network speed issues that seem to come out of nowhere? Perhaps you can't connect to certain machines from other virtual machines?

Cause:

There is a bug related VMware network adapters or their drivers related to "segmentation offload" (TSO and GSO). I have heard one mention that the checksum feature may also have problems.

Solution:

Turning off TSO/ESO on the guest OSes is the fool-proof solution. You could try installing a newer OS or try playing with the adapter type (eg: E1000, VMxnet2) but that seems to work only some of the time.

What is TSO:

TSO and GSO (standing for tcp segmentation offload and generic segmentation offload) means that instead of the CPU breaking up packets, the network card is expected to do it.

Consequences of turning it off:

Having it on improves performance by reducing CPU load. Many reports say that they don't experience CPU speed issues until they reach ridiculous amounts of sustained data (>50Mbps). Other than that, it will speed up performance to disable it according to many, and certainly will if you are grinding to a halt due to this TSO bug.

 

Test It Out:

On Unix, you can usually check if TSO and GSO are on by running

 

ethtool -k eth0

and you can disable them it by running

ethtool -K eth0 tso off gso off

If it says "operation not supported", you may want to still try setting only the TSO or only the GSO.

If ethtool isn't installed, do the following:

apt-get install ethtool

 

On Windows, go to 

Network Connection icon -> Properties -> Configure -> Advanced Tab.

Look for "TsoEnable", "LargeSendOffload", "IPv4 TSO Offload" or otherwise and set it to 0 / Disabled.

You may also want to look for things like "IPv4 Giant TSO Offload", I turn them off to be safe but I haven't actually confirmed if they matter or not. Look over and check for any other Tso related settings too.

Do It On Reboot For Unix:

There are many possible ways of doing this. You may think to try adding a line to /etc/network/interfaces, changing /etc/networking/if-up.d/ethtool, or even changing ETHTOOL_OPTS= that is set elsewhere and passed into if.up.d/ethtool. Those ways allow may seem to work after an ifup but they may fail during bootup because there may or may not be other steps (like with vmware-tools) in the bootup process that change the TSO back on.

To find the offending places, you may run:

 

grep -ri "ethtool" /etc/* | more

Since we prefer to see the settings easily in the same place you would check your IP and other settings, we went with this 2 step approach:

  1. Comment out or change any lines that the above grep shows to change tso to on. Often this may be /etc/init.d/vmware-tools.
  2. In case the above is not affecting all the nics or vmware-tools is not installed or otherwise doesn't have any lines for TSO on and the default is to have it on, we still add this to /etc/network/interfaces:

 

up ethtool -K eth0 tso off gso off

 

It also serves as a great reminder if you go there to check network settings and notice it missing.

If You Found This Pleasant Solutions Blog Entry Useful

As always, if you found this article useful and you have a blog of your own, a quick mention would be always be appreciated. If you saw another place where people were asking for this information and did not have an answer yet, helping them out would also be good.

References:

 Vmware Specific:

 

Not-Vmware Specific:

 

Unix Related: