TCP stalls with 802.3ad + bridge + kvm guest

TCP stalls with 802.3ad + bridge + kvm guest

[Peter Samuelson]

So, we have the following network stack:

    ixgbe [10 Gbit port] -- bonding [802.3ad] -- bridge -- KVM guest

(There's also a VLAN layer, but I can reproduce this problem without
it.)  It all works, except that with some flows in the KVM guest - I
can reproduce using smbclient - transfers keep stalling, such that I'm
averaging well under 1 MB/s.  Should be more like 100 MB/s.

Oddly, this only occurs when both the 802.3ad and KVM are used:

    Server        Agg        Client         TCP stalls
    --------------------------------------------------
    external      none       KVM guest      no
    external      802.3ad    KVM host       no
    KVM host      802.3ad    KVM guest      no
    external      802.3ad    KVM guest      yes

I don't understand the stalls.  'ping -f' does not show any dropped
packets.  tcpdump seems to show a lot of retransmits (server to
client), out-of-order TCP segments (server to client), and duplicate
ACKs (client to server).

Further notes:

- OS for KVM host (and guest) is Debian stable, with kernels from
  Debian backports.  I've tried several kernels including 3.4,
  currently using 3.2.20.

- Arista 10 Gbit switch, no congestion to speak of, all the test
  traffic is local to the switch.

- I can reproduce with either 1 or 2 active ports in the LACP group.

- The host IP is bound to the bridge, not directly to bond0.

- First noticed problem with a Windows VM and SMB.  I can reproduce
  100% using smbclient, but wget (http) goes full speed.

Does any of this sound familiar?  Is it a known issue?  Can anyone
offer any hints?  I can run tcpdump on the client, the server or any
point in the KVM host network stack, in case anyone is better at
interpreting them than I am.

Thanks,
Peter

Re: TCP stalls with 802.3ad + bridge + kvm guest

[Jay Vosburgh]

Peter Samuelson <psamuelson@efolder.net> wrote:
>So, we have the following network stack:
>
>    ixgbe [10 Gbit port] -- bonding [802.3ad] -- bridge -- KVM guest
>
>(There's also a VLAN layer, but I can reproduce this problem without
>it.)  It all works, except that with some flows in the KVM guest - I
>can reproduce using smbclient - transfers keep stalling, such that I'm
>averaging well under 1 MB/s.  Should be more like 100 MB/s.
>
>Oddly, this only occurs when both the 802.3ad and KVM are used:
>
>    Server        Agg        Client         TCP stalls
>    --------------------------------------------------
>    external      none       KVM guest      no
>    external      802.3ad    KVM host       no
>    KVM host      802.3ad    KVM guest      no
>    external      802.3ad    KVM guest      yes

	Does the "none" for Agg (the first line) mean no bonding at all?

	Does the problem happen if the bond is a different mode
(balance-xor, for example)?

>I don't understand the stalls.  'ping -f' does not show any dropped
>packets.  tcpdump seems to show a lot of retransmits (server to
>client), out-of-order TCP segments (server to client), and duplicate
>ACKs (client to server).

	Do the various stats on the host and guest show any drops?
E.g., from "netstat -i" and "tc -s qdisc"

>Further notes:
>
>- OS for KVM host (and guest) is Debian stable, with kernels from
>  Debian backports.  I've tried several kernels including 3.4,
>  currently using 3.2.20.
>
>- Arista 10 Gbit switch, no congestion to speak of, all the test
>  traffic is local to the switch.
>
>- I can reproduce with either 1 or 2 active ports in the LACP group.
>
>- The host IP is bound to the bridge, not directly to bond0.
>
>- First noticed problem with a Windows VM and SMB.  I can reproduce
>  100% using smbclient, but wget (http) goes full speed.
>
>Does any of this sound familiar?  Is it a known issue?  Can anyone
>offer any hints?  I can run tcpdump on the client, the server or any
>point in the KVM host network stack, in case anyone is better at
>interpreting them than I am.

	Maybe; I've seen a similar-sounding problem with CIFS wherein
the loss of the last or near-last packet that's part of the CIFS request
will cause TCP to run a full RTO.  This occurs because CIFS has no more
packets to send, as it's waiting for a response, so there is no
subsequent traffic that will trigger duplicate ACKs from the peer and
thus initiate a fast retransmission.  I may be mangling the CIFS
details, but that's the packet exchange that occurs, and it resulted in
very poor performance for CIFS.

	The case I saw this in was not using KVM, but was instead
dropping some packets at a network bottleneck.  In that case, CIFS
experienced the poor performance, but NFS did not; the NFS packet
captures also showed the lost packets, but NFS would continue to send
and issue fast retransmissions in response to the duplicate ACKs it
received.  Perhaps this mirrors your experience with CIFS vs. wget, and
your bottleneck is somewhere on the host itself in the virtual
networking.

	-J

---
	-Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com

Re: TCP stalls with 802.3ad + bridge + kvm guest

[Peter Samuelson]

> >    ixgbe [10 Gbit port] -- bonding [802.3ad] -- bridge -- KVM guest
> >
> >(There's also a VLAN layer, but I can reproduce this problem without
> >it.)  It all works, except that with some flows in the KVM guest - I
> >can reproduce using smbclient - transfers keep stalling, such that I'm
> >averaging well under 1 MB/s.  Should be more like 100 MB/s.
> >
> >Oddly, this only occurs when both the 802.3ad and KVM are used:
> >
> >    Server        Agg        Client         TCP stalls
> >    --------------------------------------------------
> >    external      none       KVM guest      no
> >    external      802.3ad    KVM host       no
> >    KVM host      802.3ad    KVM guest      no
> >    external      802.3ad    KVM guest      yes

[Jay Vosburgh]
> 	Does the "none" for Agg (the first line) mean no bonding at all?

Correct.  'None' is without the bonding driver, putting the eth
interface directly on the bridge.  (With or without a VLAN layer.)

> 	Does the problem happen if the bond is a different mode
> (balance-xor, for example)?

This is taking me longer to test, as I have to also update the switch
port config, which required coordination with coworkers.  I'll get that
info as soon as I can.

> 	Do the various stats on the host and guest show any drops?
> E.g., from "netstat -i" and "tc -s qdisc"

Yes, 'netstat -i' on the KVM host shows a few dropped RX packets on
bond0 - looks like it increments by 2 each time I do a test download of
my 20MB file.  'tc -s qdisc' on the KVM host shows nothing, and I see
no drops on the KVM guest side.

I'll get the switch reconfigured not to do LACP so I can test other
bonding modes, as you suggest.  Thanks for the quick response!  Sorry
my own followup was delayed....

Peter

Re: TCP stalls with 802.3ad + bridge + kvm guest

[Peter Samuelson]

  [Peter Samuelson]
> >    ixgbe [10 Gbit port] -- bonding [802.3ad] -- bridge -- KVM guest
> >
> >It all works, except that with some flows in the KVM guest - I
> >can reproduce using smbclient - transfers keep stalling, such that I'm
> >averaging well under 1 MB/s.  Should be more like 100 MB/s.
> >
> >Oddly, this only occurs when both the 802.3ad and KVM are used:
> >
> >    Server        Agg        Client         TCP stalls
> >    --------------------------------------------------
> >    external      none       KVM guest      no
> >    external      802.3ad    KVM host       no
> >    KVM host      802.3ad    KVM guest      no
> >    external      802.3ad    KVM guest      yes

[Jay Vosburgh]
>       Does the problem happen if the bond is a different mode
> (balance-xor, for example)?

No, the balance-xor mode seems to work normally.  I am testing it with
only a single slave interface, same way I'm testing LACP mode.

>       Do the various stats on the host and guest show any drops?
> E.g., from "netstat -i" and "tc -s qdisc"

I am not sure what to make of 'netstat -i'.  Some test boots, I get
26000 or so RX packets dropped on bond0 and eth2 (the slave interface),
but not during the actual file transfer tests.  The current test boot,
with LACP enabled, 'netstat -i' shows only 19 dropped RX packets from
bond0 (compared to 740k total RX packets), after several test runs.

Thanks,
Peter

Re: TCP stalls with 802.3ad + bridge + kvm guest

[Jesse Brandeburg]

On Wed, Aug 1, 2012 at 7:41 AM, Peter Samuelson <psamuelson@efolder.net> wrote:
>
>   [Peter Samuelson]
>> >    ixgbe [10 Gbit port] -- bonding [802.3ad] -- bridge -- KVM guest
>> >
>> >It all works, except that with some flows in the KVM guest - I
>> >can reproduce using smbclient - transfers keep stalling, such that I'm
>> >averaging well under 1 MB/s.  Should be more like 100 MB/s.
>> >
>> >Oddly, this only occurs when both the 802.3ad and KVM are used:

is it because offloads are enabled?  I had to disable several of them
to get my virtual machines to work on both igb and ixgbe interfaces, I
haven't gone any further to isolate why.

try
# ethtool -K ethx tso off gso off lro off

Re: TCP stalls with 802.3ad + bridge + kvm guest

[Peter Samuelson]

[Jesse Brandeburg]
> try
> # ethtool -K ethx tso off gso off lro off

Wow - that worked!  Thanks so much!  Now I'll see if I can narrow it
down a bit from there.

Then it's on to the mystery of why I have to explicitly set eth2+eth3
'promisc' - the bond0 device is supposed to propagate that setting but
apparently it doesn't.  (Lower priority, though, since there's an easy
workaround.)

Peter

Re: TCP stalls with 802.3ad + bridge + kvm guest

[Sławek Janecki]

I have no such info in dmsg.
I think I will file a issue to Node.js Dev Team.
Thanks for reply.

On Sat, Aug 4, 2012 at 6:22 PM, Peter Samuelson <psamuelson@efolder.net> wrote:
>
> [Jesse Brandeburg]
>> try
>> # ethtool -K ethx tso off gso off lro off
>
> Wow - that worked!  Thanks so much!  Now I'll see if I can narrow it
> down a bit from there.
>
> Then it's on to the mystery of why I have to explicitly set eth2+eth3
> 'promisc' - the bond0 device is supposed to propagate that setting but
> apparently it doesn't.  (Lower priority, though, since there's an easy
> workaround.)
>
> Peter
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 

pozdrawiam
Sławomir Janecki

Re: TCP stalls with 802.3ad + bridge + kvm guest

[Sławek Janecki]

wrong mail sorry

On Sat, Aug 4, 2012 at 8:07 PM, Sławek Janecki <janecki@gmail.com> wrote:
> I have no such info in dmsg.
> I think I will file a issue to Node.js Dev Team.
> Thanks for reply.
>
> On Sat, Aug 4, 2012 at 6:22 PM, Peter Samuelson <psamuelson@efolder.net> wrote:
>>
>> [Jesse Brandeburg]
>>> try
>>> # ethtool -K ethx tso off gso off lro off
>>
>> Wow - that worked!  Thanks so much!  Now I'll see if I can narrow it
>> down a bit from there.
>>
>> Then it's on to the mystery of why I have to explicitly set eth2+eth3
>> 'promisc' - the bond0 device is supposed to propagate that setting but
>> apparently it doesn't.  (Lower priority, though, since there's an easy
>> workaround.)
>>
>> Peter
>> --
>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
> --
>
> pozdrawiam
> Sławomir Janecki



-- 

pozdrawiam
Sławomir Janecki

Re: TCP stalls with 802.3ad + bridge + kvm guest

[Peter Samuelson]

> [Jesse Brandeburg]
> > try
> > # ethtool -K ethx tso off gso off lro off

Specifically, 'lro off' is the trick.  The others don't have a
noticeable effect.  Now I can get 70 MB/s or better most of the time,
in either direction.

Thanks again,
Peter

Re: TCP stalls with 802.3ad + bridge + kvm guest

[Stephen Hemminger]

On Sat, 4 Aug 2012 17:45:11 -0500
Peter Samuelson <psamuelson@efolder.net> wrote:

> 
> > [Jesse Brandeburg]
> > > try
> > > # ethtool -K ethx tso off gso off lro off
> 
> Specifically, 'lro off' is the trick.  The others don't have a
> noticeable effect.  Now I can get 70 MB/s or better most of the time,
> in either direction.
> 
> Thanks again,
> Peter

When device is added to bridge, the bridge code does ask the device
to turn LRO off (automatically). Unfortunately some device drivers are
broken.