Lab: v.1.8 + Linux 2.6.37.6+up #1 + ESXi 5.0 - VM - Slow Network Performance/Failures

Lab: v.1.8 + Linux 2.6.37.6+up #1 + ESXi 5.0 - VM - Slow Network Performance/Failures

[Mike Harris]

Hi,

I hope everyone is well!

Some network throughput/performance oddness with a linux based virtual firewall…

Lab scenario;

[Windows VM #1] --- VLAN X-----(*)Linux Firewall VM ----- VLAN Y
---[Windows VM #2]

A tcpdump is kicked off with the following options on the firewall
this a 100MB file is copied between VM #1 to VM #2 (SMB).

tcpdump -i eth0.x -n -s0 -w file-transfer-1.pcap -c100

Notes:

+ Physical blade run ESXi 5.0.
+ Windows VMs run on the same vSwitch and physical blade.
+ VLAN X and Y support up to 1500 bytes MTU.
+ Virtual firewall is configured with the 4095 (any) VLAN (receives
and transmits tagged frames).
+ Virtual firewall is runs Linux 2.6.37.6+up #1
+ Virtual and physical backbones do support up to 9k frames.

Observations;

1. The file copy fails…
2. The pcap reveals a 12,443 byte uber jumbo frame is present shortly
after the file transfer starts.

Repeated the same scenario using a test vyatta 6.4 VM and the file
copy completes normally… no jumbo frames or any other oddness.
Virtual and physical networking can not originate such a frame
normally.

Given this, I suspect there's a general framing failure of the network
driver on the virtual linux firewall, which lead me to the dmesg
command and this mailing list :)

Has anyone else seen this behavior before on a linux VM before?

Thoughts?

Helpful suggestions :)

Mike

Re: Lab: v.1.8 + Linux 2.6.37.6+up #1 + ESXi 5.0 - VM - Slow Network Performance/Failures

[Stephen Hemminger]

On Fri, 28 Sep 2012 04:56:35 +0200
Mike Harris <mharris@onxis.com> wrote:

> Hi,
> 
> I hope everyone is well!
> 
> Some network throughput/performance oddness with a linux based virtual firewall…
> 
> Lab scenario;
> 
> [Windows VM #1] --- VLAN X-----(*)Linux Firewall VM ----- VLAN Y
> ---[Windows VM #2]
> 
> A tcpdump is kicked off with the following options on the firewall
> this a 100MB file is copied between VM #1 to VM #2 (SMB).
> 
> tcpdump -i eth0.x -n -s0 -w file-transfer-1.pcap -c100
> 
> Notes:
> 
> + Physical blade run ESXi 5.0.
> + Windows VMs run on the same vSwitch and physical blade.
> + VLAN X and Y support up to 1500 bytes MTU.
> + Virtual firewall is configured with the 4095 (any) VLAN (receives
> and transmits tagged frames).
> + Virtual firewall is runs Linux 2.6.37.6+up #1
> + Virtual and physical backbones do support up to 9k frames.
> 
> Observations;
> 
> 1. The file copy fails…
> 2. The pcap reveals a 12,443 byte uber jumbo frame is present shortly
> after the file transfer starts.
> 
> Repeated the same scenario using a test vyatta 6.4 VM and the file
> copy completes normally… no jumbo frames or any other oddness.
> Virtual and physical networking can not originate such a frame
> normally.
> 
> Given this, I suspect there's a general framing failure of the network
> driver on the virtual linux firewall, which lead me to the dmesg
> command and this mailing list :)
> 
> Has anyone else seen this behavior before on a linux VM before?
> 
> Thoughts?
> 
> Helpful suggestions :)

Vyatta ran into a problem because the Vmware driver was incorrectly keeping LRO
enabled even when forwarding. Given the age of the kernel that could
be your problem.

The short term workaround used to just force LRO off (change to vmxnet3).
Thankfully, someone later found where the driver was mistakenly renabling LRO
and fixed the real bug.