Poor NIC performance on domUs / Crashing

Poor NIC performance on domUs / Crashing

[ryan]

Hi All, 

Setup is as follows:

Hardware: 

AMD Sempron 2800+ (No VT) 
3GB RAM 
8x250GB IDE Drives in RAID5 

Software
Xen-3.4.1 

dom0: Gentoo built with Sources 2.6.29-r4 
domU: Gentoo built with Sources 2.6.29-r4

dom0 is running as my firewall while domU is running mail/web services.
Here's my dd tests on both the dom0 and domU:


dom0:

fs ~ # time dd if=/dev/sda of=/dev/null bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 4.71219 s, 111 MB/s

real	0m4.718s
user	0m0.000s
sys	0m1.844s

fs ~ # time hdparm -t /dev/sda

/dev/sda:
 Timing buffered disk reads:  598 MB in  3.19 seconds = 187.22 MB/sec

real	0m6.598s
user	0m0.131s
sys	0m2.059s


domU:

mail ~ # time dd if=/dev/xvda of=/dev/null bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 9.39719 s, 55.8 MB/s

real	0m9.404s
user	0m0.000s
sys	0m1.317s

mail ~ # time hdparm -t /dev/xvda 

/dev/xvda:
 Timing buffered disk reads:  568 MB in  3.00 seconds = 189.19 MB/sec

real	0m6.161s
user	0m0.121s
sys	0m1.789s

Here's my config file for the domU:

fs ~ # cat /etc/xen/mail
# general
name    = "mail";
memory  = 1024;

# booting
kernel  = "/mnt/data/vms/xen/kernels/vmlinuz-2.6.29-r4-xenU";

# virtual harddisk
disk = [ "file:/mnt/data/vms/mail/domU-mail,xvda,w" ];
root = "/dev/xvda ro";

# virtual network
vif = [ "ip=10.9.18.13, vifname=eth3" ];
#vif = [ "" ];
#dhcp = "dhcp";

# If you can't see anything from init in your "xm console" command try
uncommenting this line.
extra = 'xencons=tty'


When I run rsync from a network host into the domU my pings to the domU go
to 1000ms and my write speed is around 1MB/s. I've already set the ethtool
-K eth0 tx off on the domU:

mail ~ # ethtool -k eth0
Offload parameters for eth0:
Cannot get device rx csum settings: Operation not supported
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off


Also, I've had the whole kernel crash on the dom0 when doing moderate
network activity to the domU. Got some message about insufficient memory on
the eth0 interface (WAN facing on the dom0).

Any ideas as to how to troubleshoot this?

Thanks!
Ryan

Re: Poor NIC performance on domUs / Crashing

[Jeremy Fitzhardinge]

On 12/05/09 09:40, ryan@ryanholt.net wrote:
> Hi All,
>
> Setup is as follows:
>
> Hardware:
>
> AMD Sempron 2800+ (No VT)
> 3GB RAM
> 8x250GB IDE Drives in RAID5
>
> Software
> Xen-3.4.1
>
> dom0: Gentoo built with Sources 2.6.29-r4
> domU: Gentoo built with Sources 2.6.29-r4
>
> dom0 is running as my firewall while domU is running mail/web services.
> Here's my dd tests on both the dom0 and domU:
>    

Do you know what source base these kernels are based on?  Are they 
pvops?  Novell forward-port?  In particular, what's the origin of the 
2.6.29 dom0?

     J

>
> dom0:
>
> fs ~ # time dd if=/dev/sda of=/dev/null bs=1M count=500
> 500+0 records in
> 500+0 records out
> 524288000 bytes (524 MB) copied, 4.71219 s, 111 MB/s
>
> real	0m4.718s
> user	0m0.000s
> sys	0m1.844s
>
> fs ~ # time hdparm -t /dev/sda
>
> /dev/sda:
>   Timing buffered disk reads:  598 MB in  3.19 seconds = 187.22 MB/sec
>
> real	0m6.598s
> user	0m0.131s
> sys	0m2.059s
>
>
> domU:
>
> mail ~ # time dd if=/dev/xvda of=/dev/null bs=1M count=500
> 500+0 records in
> 500+0 records out
> 524288000 bytes (524 MB) copied, 9.39719 s, 55.8 MB/s
>
> real	0m9.404s
> user	0m0.000s
> sys	0m1.317s
>
> mail ~ # time hdparm -t /dev/xvda
>
> /dev/xvda:
>   Timing buffered disk reads:  568 MB in  3.00 seconds = 189.19 MB/sec
>
> real	0m6.161s
> user	0m0.121s
> sys	0m1.789s
>
> Here's my config file for the domU:
>
> fs ~ # cat /etc/xen/mail
> # general
> name    = "mail";
> memory  = 1024;
>
> # booting
> kernel  = "/mnt/data/vms/xen/kernels/vmlinuz-2.6.29-r4-xenU";
>
> # virtual harddisk
> disk = [ "file:/mnt/data/vms/mail/domU-mail,xvda,w" ];
> root = "/dev/xvda ro";
>
> # virtual network
> vif = [ "ip=10.9.18.13, vifname=eth3" ];
> #vif = [ "" ];
> #dhcp = "dhcp";
>
> # If you can't see anything from init in your "xm console" command try
> uncommenting this line.
> extra = 'xencons=tty'
>
>
> When I run rsync from a network host into the domU my pings to the domU go
> to 1000ms and my write speed is around 1MB/s. I've already set the ethtool
> -K eth0 tx off on the domU:
>
> mail ~ # ethtool -k eth0
> Offload parameters for eth0:
> Cannot get device rx csum settings: Operation not supported
> rx-checksumming: off
> tx-checksumming: off
> scatter-gather: off
> tcp segmentation offload: off
> udp fragmentation offload: off
> generic segmentation offload: off
>
>
> Also, I've had the whole kernel crash on the dom0 when doing moderate
> network activity to the domU. Got some message about insufficient memory on
> the eth0 interface (WAN facing on the dom0).
>
> Any ideas as to how to troubleshoot this?
>
> Thanks!
> Ryan
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
>    

Re: Poor NIC performance on domUs / Crashing

[Andrew Lyon]

On Fri, Dec 11, 2009 at 6:24 PM, Jeremy Fitzhardinge <jeremy@goop.org> wrote:
> On 12/05/09 09:40, ryan@ryanholt.net wrote:
>>
>> Hi All,
>>
>> Setup is as follows:
>>
>> Hardware:
>>
>> AMD Sempron 2800+ (No VT)
>> 3GB RAM
>> 8x250GB IDE Drives in RAID5
>>
>> Software
>> Xen-3.4.1
>>
>> dom0: Gentoo built with Sources 2.6.29-r4
>> domU: Gentoo built with Sources 2.6.29-r4
>>
>> dom0 is running as my firewall while domU is running mail/web services.
>> Here's my dd tests on both the dom0 and domU:
>>
>
> Do you know what source base these kernels are based on?  Are they pvops?
>  Novell forward-port?  In particular, what's the origin of the 2.6.29 dom0?

>From the description I believe they are the rebased novell forward
ported patches which I rebased to apply to vanilla without other
novell/suse patches and made available for download at
http://code.google.com/p/gentoo-xen-kernel/downloads/list



>
>    J
>
>>
>> dom0:
>>
>> fs ~ # time dd if=/dev/sda of=/dev/null bs=1M count=500
>> 500+0 records in
>> 500+0 records out
>> 524288000 bytes (524 MB) copied, 4.71219 s, 111 MB/s
>>
>> real    0m4.718s
>> user    0m0.000s
>> sys     0m1.844s
>>
>> fs ~ # time hdparm -t /dev/sda
>>
>> /dev/sda:
>>  Timing buffered disk reads:  598 MB in  3.19 seconds = 187.22 MB/sec
>>
>> real    0m6.598s
>> user    0m0.131s
>> sys     0m2.059s
>>
>>
>> domU:
>>
>> mail ~ # time dd if=/dev/xvda of=/dev/null bs=1M count=500
>> 500+0 records in
>> 500+0 records out
>> 524288000 bytes (524 MB) copied, 9.39719 s, 55.8 MB/s
>>
>> real    0m9.404s
>> user    0m0.000s
>> sys     0m1.317s
>>
>> mail ~ # time hdparm -t /dev/xvda
>>
>> /dev/xvda:
>>  Timing buffered disk reads:  568 MB in  3.00 seconds = 189.19 MB/sec
>>
>> real    0m6.161s
>> user    0m0.121s
>> sys     0m1.789s
>>
>> Here's my config file for the domU:
>>
>> fs ~ # cat /etc/xen/mail
>> # general
>> name    = "mail";
>> memory  = 1024;
>>
>> # booting
>> kernel  = "/mnt/data/vms/xen/kernels/vmlinuz-2.6.29-r4-xenU";
>>
>> # virtual harddisk
>> disk = [ "file:/mnt/data/vms/mail/domU-mail,xvda,w" ];
>> root = "/dev/xvda ro";
>>
>> # virtual network
>> vif = [ "ip=10.9.18.13, vifname=eth3" ];
>> #vif = [ "" ];
>> #dhcp = "dhcp";
>>
>> # If you can't see anything from init in your "xm console" command try
>> uncommenting this line.
>> extra = 'xencons=tty'
>>
>>
>> When I run rsync from a network host into the domU my pings to the domU go
>> to 1000ms and my write speed is around 1MB/s. I've already set the ethtool
>> -K eth0 tx off on the domU:
>>
>> mail ~ # ethtool -k eth0
>> Offload parameters for eth0:
>> Cannot get device rx csum settings: Operation not supported
>> rx-checksumming: off
>> tx-checksumming: off
>> scatter-gather: off
>> tcp segmentation offload: off
>> udp fragmentation offload: off
>> generic segmentation offload: off
>>
>>
>> Also, I've had the whole kernel crash on the dom0 when doing moderate
>> network activity to the domU. Got some message about insufficient memory
>> on
>> the eth0 interface (WAN facing on the dom0).
>>
>> Any ideas as to how to troubleshoot this?
>>
>> Thanks!
>> Ryan
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>
>>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

RE: Poor NIC performance on domUs / Crashing

[Ryan Holt]

Hi Andrew, Jeremy,

Andrew is right. I used the sources listed there.

The domU seems to run fine so long as I don't push any high amount of data
to it. As soon as I start an FTP transfer the ping milliseconds jumps up to
1000ms.

Thanks,
Ryan

-----Original Message-----
From: Andrew Lyon [mailto:andrew.lyon@gmail.com] 
Sent: Friday, December 11, 2009 2:17 PM
To: Jeremy Fitzhardinge
Cc: ryan@ryanholt.net; xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] Poor NIC performance on domUs / Crashing

On Fri, Dec 11, 2009 at 6:24 PM, Jeremy Fitzhardinge <jeremy@goop.org>
wrote:
> On 12/05/09 09:40, ryan@ryanholt.net wrote:
>>
>> Hi All,
>>
>> Setup is as follows:
>>
>> Hardware:
>>
>> AMD Sempron 2800+ (No VT)
>> 3GB RAM
>> 8x250GB IDE Drives in RAID5
>>
>> Software
>> Xen-3.4.1
>>
>> dom0: Gentoo built with Sources 2.6.29-r4
>> domU: Gentoo built with Sources 2.6.29-r4
>>
>> dom0 is running as my firewall while domU is running mail/web services.
>> Here's my dd tests on both the dom0 and domU:
>>
>
> Do you know what source base these kernels are based on?  Are they pvops?
>  Novell forward-port?  In particular, what's the origin of the 2.6.29
dom0?

>From the description I believe they are the rebased novell forward
ported patches which I rebased to apply to vanilla without other
novell/suse patches and made available for download at
http://code.google.com/p/gentoo-xen-kernel/downloads/list



>
>    J
>
>>
>> dom0:
>>
>> fs ~ # time dd if=/dev/sda of=/dev/null bs=1M count=500
>> 500+0 records in
>> 500+0 records out
>> 524288000 bytes (524 MB) copied, 4.71219 s, 111 MB/s
>>
>> real    0m4.718s
>> user    0m0.000s
>> sys     0m1.844s
>>
>> fs ~ # time hdparm -t /dev/sda
>>
>> /dev/sda:
>>  Timing buffered disk reads:  598 MB in  3.19 seconds = 187.22 MB/sec
>>
>> real    0m6.598s
>> user    0m0.131s
>> sys     0m2.059s
>>
>>
>> domU:
>>
>> mail ~ # time dd if=/dev/xvda of=/dev/null bs=1M count=500
>> 500+0 records in
>> 500+0 records out
>> 524288000 bytes (524 MB) copied, 9.39719 s, 55.8 MB/s
>>
>> real    0m9.404s
>> user    0m0.000s
>> sys     0m1.317s
>>
>> mail ~ # time hdparm -t /dev/xvda
>>
>> /dev/xvda:
>>  Timing buffered disk reads:  568 MB in  3.00 seconds = 189.19 MB/sec
>>
>> real    0m6.161s
>> user    0m0.121s
>> sys     0m1.789s
>>
>> Here's my config file for the domU:
>>
>> fs ~ # cat /etc/xen/mail
>> # general
>> name    = "mail";
>> memory  = 1024;
>>
>> # booting
>> kernel  = "/mnt/data/vms/xen/kernels/vmlinuz-2.6.29-r4-xenU";
>>
>> # virtual harddisk
>> disk = [ "file:/mnt/data/vms/mail/domU-mail,xvda,w" ];
>> root = "/dev/xvda ro";
>>
>> # virtual network
>> vif = [ "ip=10.9.18.13, vifname=eth3" ];
>> #vif = [ "" ];
>> #dhcp = "dhcp";
>>
>> # If you can't see anything from init in your "xm console" command try
>> uncommenting this line.
>> extra = 'xencons=tty'
>>
>>
>> When I run rsync from a network host into the domU my pings to the domU
go
>> to 1000ms and my write speed is around 1MB/s. I've already set the
ethtool
>> -K eth0 tx off on the domU:
>>
>> mail ~ # ethtool -k eth0
>> Offload parameters for eth0:
>> Cannot get device rx csum settings: Operation not supported
>> rx-checksumming: off
>> tx-checksumming: off
>> scatter-gather: off
>> tcp segmentation offload: off
>> udp fragmentation offload: off
>> generic segmentation offload: off
>>
>>
>> Also, I've had the whole kernel crash on the dom0 when doing moderate
>> network activity to the domU. Got some message about insufficient memory
>> on
>> the eth0 interface (WAN facing on the dom0).
>>
>> Any ideas as to how to troubleshoot this?
>>
>> Thanks!
>> Ryan
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>
>>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: Poor NIC performance on domUs / Crashing

[Jeremy Fitzhardinge]

On 12/11/2009 11:20 AM, Ryan Holt wrote:
> Andrew is right. I used the sources listed there.
>
> The domU seems to run fine so long as I don't push any high amount of data
> to it. As soon as I start an FTP transfer the ping milliseconds jumps up to
> 1000ms.
>    

It sounds rather like a bug that was fixed a long time ago in 2.6.18-xen 
and more recently in pvops dom0.  But you'll need to talk to Gentoo to 
get support for those kernels; they are many steps removed from the Xen 
project.

     J

Re: Poor NIC performance on domUs / Crashing

[Ryan Holt]

There another kernel source you'd recommend using?

Sent from my iPhone

On Dec 11, 2009, at 4:51 PM, Jeremy Fitzhardinge <jeremy@goop.org>  
wrote:

> On 12/11/2009 11:20 AM, Ryan Holt wrote:
>> Andrew is right. I used the sources listed there.
>>
>> The domU seems to run fine so long as I don't push any high amount  
>> of data
>> to it. As soon as I start an FTP transfer the ping milliseconds  
>> jumps up to
>> 1000ms.
>>
>
> It sounds rather like a bug that was fixed a long time ago in 2.6.18- 
> xen and more recently in pvops dom0.  But you'll need to talk to  
> Gentoo to get support for those kernels; they are many steps removed  
> from the Xen project.
>
>    J

Re: Poor NIC performance on domUs / Crashing

[Jeremy Fitzhardinge]

On 12/11/2009 03:21 PM, Ryan Holt wrote:
> There another kernel source you'd recommend using?

http://wiki.xensource.com/xenwiki/XenParavirtOps

     J

Re: Poor NIC performance on domUs / Crashing

[Ryan Holt]

So is this an issue with the dom0 or the domU?

Jeremy Fitzhardinge wrote:
> On 12/11/2009 03:21 PM, Ryan Holt wrote:
>> There another kernel source you'd recommend using?
>
> http://wiki.xensource.com/xenwiki/XenParavirtOps
>
>     J

Re: Poor NIC performance on domUs / Crashing

[Jeremy Fitzhardinge]

On 12/11/2009 04:13 PM, Ryan Holt wrote:
> So is this an issue with the dom0 or the domU?
>    
The bug I'm referring to was in netback in dom0, where it did the wrong 
thing with gso packets.

     J

Re: Poor NIC performance on domUs / Crashing

[Ryan Holt]

Thanks Jeremy.

I installed xen-sources-2.6.18-r12 as my dom0 and the issue seems to
have gone away. Will do some more thorough throughput testing on it
later. I'm using gentoo-sources-2.6.31-r6 for the domU.

Question; what's the typical type of performance decrease I should
expect to the domU over the network? Getting around 12MB/sec to the
dom0, and anywhere from 5-10MB/sec to the domU; that typical/expected?

Thanks,
Ryan

Jeremy Fitzhardinge wrote:
> On 12/11/2009 04:13 PM, Ryan Holt wrote:
>> So is this an issue with the dom0 or the domU?
>>    
> The bug I'm referring to was in netback in dom0, where it did the
> wrong thing with gso packets.
>
>     J

Re: Poor NIC performance on domUs / Crashing

[Jeremy Fitzhardinge]

On 12/12/09 09:10, Ryan Holt wrote:
> I installed xen-sources-2.6.18-r12 as my dom0 and the issue seems to
> have gone away. Will do some more thorough throughput testing on it
> later. I'm using gentoo-sources-2.6.31-r6 for the domU.
>
> Question; what's the typical type of performance decrease I should
> expect to the domU over the network? Getting around 12MB/sec to the
> dom0, and anywhere from 5-10MB/sec to the domU; that typical/expected?
>    

That sounds reasonable to me, but I haven't done much 
performance-oriented testing so I don't know whether that's really 
expected.  There are a lot of variables in there..

     J