Re: vxlan/veth performance issues on net.git + latest kernels

[Joseph Gasparakis <joseph.gasparakis@intel.com>]

On Sat, 7 Dec 2013, Or Gerlitz wrote:

> On Fri, Dec 6, 2013, Joseph Gasparakis <joseph.gasparakis@intel.com> wrote:
> > On Fri, 6 Dec 2013, Or Gerlitz wrote:
> >
> >> On 04/12/2013 11:41, Or Gerlitz wrote:
> >> > On Wed, Dec 4, 2013 at 11:24 AM, Joseph
> >> > Gasparakis<joseph.gasparakis@intel.com>  wrote:
> >> > > >And just for the record,
> >> > > >you are seeing (SKB_UDP_TUNNEL | SKB_GSO_TCPV4) as 0x201 while I was
> >> > > >seeing it as 0x81 because commit 61c1db7fae "ipv6: sit: add GSO/TSO
> >> > > >support" pushed the SKB_UDP_TUNNEL two bits left, and I had done my tests
> >> > > before it.
> >> > indeed, also, on what kernel did you conducted your tests which you managed
> >> > to WA the problem with unsetting that bit?
> >>
> >>
> >> Hi Joseph,
> >>
> >> Really need your response here --
> >
> > I'm sorry Or, I managed to miss your original request...
> 
> sure.. it happens.
> 
> >> 1. on which kernel did you manage to get along fine vxlan performance wise
> >> with this hack?
> 
> > I was running 3.10.6.
> 
> I see, will try it out, and just for getting closer to your env, what
> kernel where the guests running? was a bridge / ovs instance involved
> in the VM PV connectivity?
>

The VMs were running an old 2.6.32 kernel, although if I remember well I 
also tried a 3.x - sorry can't remember which one. The VMs were attached 
to a bridge, but I haven't noticed any packet loss there.
 
> 
> >> 2. did the hack helped for both veth host traffic or only on PV VM traffic?
> 
> > No, just VM. I haven't tried veth.
> 
> I see, earlier I was somehow under the impression you noted the
> problem for veth too.
> 
> > If you leave the DODGY bit, does your traffic get droped on Tx, after it
> > leaves vxlan and before it hits your driver, which is what I had seen. Is
> > that right?
>
> What I saw is that if I leave the DODGY bit set, practically things
> don't work at all, its not that some packets are dropped, was that
> what you saw?
>
What I saw was gso packets badly segmented, causing many re-transmissions 
and dropping the performance to a few MB/s.
 
> Or on your env only **some** or **few** packets were dropped each time
> but this killed the tcp session performance?
> 
> Also, did you hack/modified the VM NIC MTU to take into the account
> the encapsulation overhead?
> 
The virtio interfaces I used had MTU 1500, but the MTU of the physical NIC 
was increased to 1600.

> > If you unset it, do you recover?
> 
> let me redo this with your setting and see, please make sure to tell
> me what kernel the VM was running too (thanks!)
> 
> > What is the output of your ethtool -k on the interface you are
> > transmitting from?
> 
> will send you tomorrow, but this happens without offloads for
> encapsulated traffic.
> 
I have only noticed this with the offloads on. Turning off encapsuation 
TSO off, would simply make the gso's to get segmented in dev_hard_xmit() 
as expected.