domU network slowdown with new dom0 kernel

domU network slowdown with new dom0 kernel

[Denis Chapligin]

Hi

I have a domU that acts as iSCSI target for our network. It was running
on the host with xen 3.4.2 and original XenLinux (2.6.18) kernel. Today
i've tried to replace old 2.6.18 kernel with a latest (stable/master)
pvops kernel. The problem, is that when i replaced dom0 kernel, network
performance of domU mentioned above dropped and networking with this
domU become totaly unusable. In domU's dmesg i see the following
entries:

[ 1785.260012] net eth0: rx->offset: 0, size: 4294967295
[ 1785.260036] net eth0: rx->offset: 0, size: 4294967295
[ 1785.810612] net eth0: rx->offset: 0, size: 4294967295
[ 1785.810612] net eth0: rx->offset: 0, size: 4294967295
[ 1786.310061] net eth0: rx->offset: 0, size: 4294967295
[ 1786.310081] net eth0: rx->offset: 0, size: 4294967295
[ 1786.550007] net eth0: rx->offset: 0, size: 4294967295
[ 1786.550024] net eth0: rx->offset: 0, size: 4294967295
[ 1786.970010] net eth0: rx->offset: 0, size: 4294967295
[ 1786.970025] net eth0: rx->offset: 0, size: 4294967295
[ 1792.840010] __ratelimit: 6 callbacks suppressed
[ 1792.840035] net eth0: rx->offset: 0, size: 4294967295
[ 1792.840049] net eth0: rx->offset: 0, size: 4294967295
[ 1798.310016] net eth0: rx->offset: 0, size: 4294967295
[ 1798.310045] net eth0: rx->offset: 0, size: 4294967295
[ 1799.230013] net eth0: rx->offset: 0, size: 4294967295
[ 1799.230036] net eth0: rx->offset: 0, size: 4294967295
[ 1799.550009] net eth0: rx->offset: 0, size: 4294967295
[ 1799.550031] net eth0: rx->offset: 0, size: 4294967295
[ 1800.110352] net eth0: rx->offset: 0, size: 4294967295
[ 1800.110352] net eth0: rx->offset: 0, size: 4294967295
[ 1800.350020] net eth0: rx->offset: 0, size: 4294967295
[ 1800.350050] net eth0: rx->offset: 0, size: 4294967295

in dom0's dmesg i see nothing interesting. I've tried several kernels
and found that domU could use any kernel, but dom0 kernel version is
critical and it only works with xen 2.6.18 kernel.

I have 6 xen hosts, but only this domU is problematic. The domU above is
very high loaded and installed on host with 4x1GB ethernet card,
combined into single etherchannel. All other domU's on other hosts
and this host aren't affected and use only 1 NIC and they're
bandwidth usage usually not more than 10-15% of that NIC. The
problematic domU uses up to 80% of combined bandwidth.

So, what i'm doing wrong and how can i fix this issue? :)

-- 
						Denis Chapligin

Re: domU network slowdown with new dom0 kernel

[Jeremy Fitzhardinge]

On 03/01/2010 04:57 AM, Denis Chapligin wrote:
> Hi
>
> I have a domU that acts as iSCSI target for our network. It was running
> on the host with xen 3.4.2 and original XenLinux (2.6.18) kernel. Today
> i've tried to replace old 2.6.18 kernel with a latest (stable/master)
> pvops kernel. The problem, is that when i replaced dom0 kernel, network
> performance of domU mentioned above dropped and networking with this
> domU become totaly unusable. In domU's dmesg i see the following
> entries:
>
> [ 1785.260012] net eth0: rx->offset: 0, size: 4294967295
> [ 1785.260036] net eth0: rx->offset: 0, size: 4294967295
> [ 1785.810612] net eth0: rx->offset: 0, size: 4294967295
> [ 1785.810612] net eth0: rx->offset: 0, size: 4294967295
> [ 1786.310061] net eth0: rx->offset: 0, size: 4294967295
> [ 1786.310081] net eth0: rx->offset: 0, size: 4294967295
> [ 1786.550007] net eth0: rx->offset: 0, size: 4294967295
> [ 1786.550024] net eth0: rx->offset: 0, size: 4294967295
> [ 1786.970010] net eth0: rx->offset: 0, size: 4294967295
> [ 1786.970025] net eth0: rx->offset: 0, size: 4294967295
> [ 1792.840010] __ratelimit: 6 callbacks suppressed
> [ 1792.840035] net eth0: rx->offset: 0, size: 4294967295
> [ 1792.840049] net eth0: rx->offset: 0, size: 4294967295
> [ 1798.310016] net eth0: rx->offset: 0, size: 4294967295
> [ 1798.310045] net eth0: rx->offset: 0, size: 4294967295
> [ 1799.230013] net eth0: rx->offset: 0, size: 4294967295
> [ 1799.230036] net eth0: rx->offset: 0, size: 4294967295
> [ 1799.550009] net eth0: rx->offset: 0, size: 4294967295
> [ 1799.550031] net eth0: rx->offset: 0, size: 4294967295
> [ 1800.110352] net eth0: rx->offset: 0, size: 4294967295
> [ 1800.110352] net eth0: rx->offset: 0, size: 4294967295
> [ 1800.350020] net eth0: rx->offset: 0, size: 4294967295
> [ 1800.350050] net eth0: rx->offset: 0, size: 4294967295
>
> in dom0's dmesg i see nothing interesting. I've tried several kernels
> and found that domU could use any kernel, but dom0 kernel version is
> critical and it only works with xen 2.6.18 kernel.
>
> I have 6 xen hosts, but only this domU is problematic. The domU above is
> very high loaded and installed on host with 4x1GB ethernet card,
> combined into single etherchannel. All other domU's on other hosts
> and this host aren't affected and use only 1 NIC and they're
> bandwidth usage usually not more than 10-15% of that NIC. The
> problematic domU uses up to 80% of combined bandwidth.
>
> So, what i'm doing wrong and how can i fix this issue? :)
>
>    

Is this with a freshly checked out version of xen.git?  I committed some 
bugfixes to netback last week, which may have caused problems.

     J