Serious performance issues

Serious performance issues

[Tommi Virtanen]

Hi. I have a Shuttle box with an AMD Athlon XP 2200+ and 1GB of RAM.
I'm normally running it with Debian sarge/sid and kernel 2.6.10-1-k7,
as built by Debian. I want to use Xen on it. I built a xen0 kernel
which is as close to the Debian kernel as I can (no power management,
no HPET timers, broken ISA drivers disabled), disabled /lib/tls, and
booted with the new kernel.

Everything works. I can run domUs as I want. But, the performance is
horrible, even with just dom0 running.

I tested the disk, it still transfers ~50MB/s. I don't seem to be
suffering from PIO mode. Also, things running from RAM cache are as
slow as from disk.

I ran some of the lmbench tests both under xen and with native linux.
Here are the results, with the "-" lines from xen and "+" lines from
native linux. The results are pretty poor, and so is the feel of the
machine under xen.

Any hints on what to try next?


bw_mem
-0.131072 1068.95
+0.131072 4409.00

bw_pipe
-Pipe bandwidth: 85.42 MB/sec
+Pipe bandwidth: 989.19 MB/sec

bw_unix
-AF_UNIX sock stream bandwidth: 104.75 MB/sec
+AF_UNIX sock stream bandwidth: 968.87 MB/sec

lat_ctx
-"size=0k ovr=3.15
-10 13.95
+"size=0k ovr=1.09
+10 0.94

lat_fcntl
-Fcntl lock latency: 43.2246 microseconds
+Fcntl lock latency: 5.4676 microseconds

lat_fifo
-FIFO latency: 32.4076 microseconds
+FIFO latency: 3.3622 microseconds

lat_fs
-0k     1000    18563   31549
-1k     1000    6426    17887
-4k     1000    6583    17826
-10k    1000    3084    14989
+0k     1000    63666   92541
+1k     1000    24125   51193
+4k     1000    25863   52089
+10k    1000    12533   41564

lat_pipe
-Pipe latency: 31.0475 microseconds
+Pipe latency: 5.4833 microseconds

lat_proc
-Process fork+exit: 789.2857 microseconds
+Process fork+exit: 113.7347 microseconds

lat_select
-Select on 200 tcp fd's: 46.8159 microseconds
+Select on 200 tcp fd's: 12.9369 microseconds

lat_syscall
-Simple syscall: 0.6188 microseconds
+Simple syscall: 0.1547 microseconds

lat_unix
-AF_UNIX sock stream latency: 61.1547 microseconds
+AF_UNIX sock stream latency: 21.7704 microseconds


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: Serious performance issues

[Anthony Liguori]

Tommi Virtanen wrote:

> I tested the disk, it still transfers ~50MB/s. I don't seem to be
> suffering from PIO mode. Also, things running from RAM cache are as
> slow as from disk.

What version of Xen is this?

> Any hints on what to try next?

If you're running xen-unstable, try xen-2.0.x or vice versa.  Are you 
absolutely positive that you've got enough memory allocated to dom0?  
That you've got the proper IO chipset support built into the kernel?  
That you've really got tls disabled (if you just move tls after an app 
has already launched using it it will still be using it until you 
restart the app)?

These numbers seem too bad to be Xen's fault alone.  Do you have swap 
enabled?  Is there a great deal of swap activity?

Regards,
Anthony Liguori

> bw_mem
> -0.131072 1068.95
> +0.131072 4409.00
>
> bw_pipe
> -Pipe bandwidth: 85.42 MB/sec
> +Pipe bandwidth: 989.19 MB/sec
>
> bw_unix
> -AF_UNIX sock stream bandwidth: 104.75 MB/sec
> +AF_UNIX sock stream bandwidth: 968.87 MB/sec
>
> lat_ctx
> -"size=0k ovr=3.15
> -10 13.95
> +"size=0k ovr=1.09
> +10 0.94
>
> lat_fcntl
> -Fcntl lock latency: 43.2246 microseconds
> +Fcntl lock latency: 5.4676 microseconds
>
> lat_fifo
> -FIFO latency: 32.4076 microseconds
> +FIFO latency: 3.3622 microseconds
>
> lat_fs
> -0k     1000    18563   31549
> -1k     1000    6426    17887
> -4k     1000    6583    17826
> -10k    1000    3084    14989
> +0k     1000    63666   92541
> +1k     1000    24125   51193
> +4k     1000    25863   52089
> +10k    1000    12533   41564
>
> lat_pipe
> -Pipe latency: 31.0475 microseconds
> +Pipe latency: 5.4833 microseconds
>
> lat_proc
> -Process fork+exit: 789.2857 microseconds
> +Process fork+exit: 113.7347 microseconds
>
> lat_select
> -Select on 200 tcp fd's: 46.8159 microseconds
> +Select on 200 tcp fd's: 12.9369 microseconds
>
> lat_syscall
> -Simple syscall: 0.6188 microseconds
> +Simple syscall: 0.1547 microseconds
>
> lat_unix
> -AF_UNIX sock stream latency: 61.1547 microseconds
> +AF_UNIX sock stream latency: 21.7704 microseconds
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
>



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

RE: Serious performance issues

[Ian Pratt]

> Hi. I have a Shuttle box with an AMD Athlon XP 2200+ and 1GB of RAM.
> I'm normally running it with Debian sarge/sid and kernel 2.6.10-1-k7,
> as built by Debian. I want to use Xen on it. I built a xen0 kernel
> which is as close to the Debian kernel as I can (no power management,
> no HPET timers, broken ISA drivers disabled), disabled /lib/tls, and
> booted with the new kernel.

> lat_syscall
> -Simple syscall: 0.6188 microseconds
> +Simple syscall: 0.1547 microseconds

System call latency should be identical between Xen and native. The fact
that it is not inidcates that your system is seriously screwed running
Xen.

Have you got some interrupt line going crazy?
Ian


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_ide95&alloc_id\x14396&op=click

RE: Serious performance issues

[Ian Pratt]

 
> Hi. I have a Shuttle box with an AMD Athlon XP 2200+ and 1GB of RAM.
> I'm normally running it with Debian sarge/sid and kernel 2.6.10-1-k7,
> as built by Debian. I want to use Xen on it. I built a xen0 kernel
> which is as close to the Debian kernel as I can (no power management,
> no HPET timers, broken ISA drivers disabled), disabled /lib/tls, and
> booted with the new kernel.

Please try using our kernel config.

BTW: you're not doing anything rash like using an SMP enabled kernel are
you? There's still loads of debugging turned on at the moment.

Ian
 
> Everything works. I can run domUs as I want. But, the performance is
> horrible, even with just dom0 running.
> 
> I tested the disk, it still transfers ~50MB/s. I don't seem to be
> suffering from PIO mode. Also, things running from RAM cache are as
> slow as from disk.
> 
> I ran some of the lmbench tests both under xen and with native linux.
> Here are the results, with the "-" lines from xen and "+" lines from
> native linux. The results are pretty poor, and so is the feel of the
> machine under xen.
> 
> Any hints on what to try next?
> 
> 
> bw_mem
> -0.131072 1068.95
> +0.131072 4409.00
> 
> bw_pipe
> -Pipe bandwidth: 85.42 MB/sec
> +Pipe bandwidth: 989.19 MB/sec
> 
> bw_unix
> -AF_UNIX sock stream bandwidth: 104.75 MB/sec
> +AF_UNIX sock stream bandwidth: 968.87 MB/sec
> 
> lat_ctx
> -"size=0k ovr=3.15
> -10 13.95
> +"size=0k ovr=1.09
> +10 0.94
> 
> lat_fcntl
> -Fcntl lock latency: 43.2246 microseconds
> +Fcntl lock latency: 5.4676 microseconds
> 
> lat_fifo
> -FIFO latency: 32.4076 microseconds
> +FIFO latency: 3.3622 microseconds
> 
> lat_fs
> -0k     1000    18563   31549
> -1k     1000    6426    17887
> -4k     1000    6583    17826
> -10k    1000    3084    14989
> +0k     1000    63666   92541
> +1k     1000    24125   51193
> +4k     1000    25863   52089
> +10k    1000    12533   41564
> 
> lat_pipe
> -Pipe latency: 31.0475 microseconds
> +Pipe latency: 5.4833 microseconds
> 
> lat_proc
> -Process fork+exit: 789.2857 microseconds
> +Process fork+exit: 113.7347 microseconds
> 
> lat_select
> -Select on 200 tcp fd's: 46.8159 microseconds
> +Select on 200 tcp fd's: 12.9369 microseconds
> 
> lat_syscall
> -Simple syscall: 0.6188 microseconds
> +Simple syscall: 0.1547 microseconds
> 
> lat_unix
> -AF_UNIX sock stream latency: 61.1547 microseconds
> +AF_UNIX sock stream latency: 21.7704 microseconds
> 
> 
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from 
> real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
> 


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_ide95&alloc_id\x14396&op=click

Re: Serious performance issues

[Tommi Virtanen]

Anthony Liguori wrote:
>> I tested the disk, it still transfers ~50MB/s. I don't seem to be
>> suffering from PIO mode. Also, things running from RAM cache are as
>> slow as from disk.
> What version of Xen is this?

2.0.4, as packaged for Debian by Adam Heath at
http://people.debian.org/~doogie/

> If you're running xen-unstable, try xen-2.0.x or vice versa.  Are you 
> absolutely positive that you've got enough memory allocated to dom0?  

As dom0 is my main environment, it gets almost all of the RAM.

title           Xen 2.6.10
root            (hd0,1)
kernel          /boot/xen.gz dom0_mem=917504
module          /boot/xen-linux-2.6.10+xen0 root=/dev/hda2 ro
module          /boot/initrd.img-2.6.10+xen0
savedefault
boot

> That you've got the proper IO chipset support built into the kernel?  

The config is pretty much identical between native linux and xen dom0.

> That you've really got tls disabled (if you just move tls after an app 
> has already launched using it it will still be using it until you 
> restart the app)?

I have rebooted many times since the move.

> These numbers seem too bad to be Xen's fault alone.  Do you have swap 
> enabled?  Is there a great deal of swap activity?

There's 2 gigs of swap with under 3 megabytes used, no blocks going in
or out.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: Serious performance issues

[Tommi Virtanen]

Ian Pratt wrote:
>>as built by Debian. I want to use Xen on it. I built a xen0 kernel
>>which is as close to the Debian kernel as I can (no power management,
> Please try using our kernel config.

I did that first, the interactive feel was as poor, and that's the
reason I went for the config that was as close to my normal native
config as possible, to try to eliminate driver issues.

> BTW: you're not doing anything rash like using an SMP enabled kernel are
> you? There's still loads of debugging turned on at the moment.

# CONFIG_SMP is not set

Will get back to you with interrupt statistics as soon as I have time
to reboot this machine.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: Serious performance issues

[Tommi Virtanen]

Ian Pratt wrote:
> Have you got some interrupt line going crazy?

That seems to be the problem.

In native linux, I see:

0	timer	1000/s
LOC		1000/s

and in xen dom0 I see:

21	uhci_hcd	35000/s
130	timer		45000/s

rmmodding uhci_hcd removes irq 21, but makes the timer
speed up to 115000/s.

rmmod ohci13943, rmmod ehci_hcd makes the timer go
800000/s.

rmmoding even more drivers puts the peak at 1100000/s,
until it suddenly drops to 30/s. Didn't check between each
rmmod, so I don't know which exact one caused it.

Unplugging my USB trackball and modprobing _all_ of the drivers above
causes timer to tick at 60/s. Plugging in my USB trackball makes it
33000/s. Unplugging has no effect.

Booting up with trackball unconnected gives about 30000/s, too, so
it's not just the USB mouse driver. It seems there is a problem with
USB.

But that doesn't explain everything. It seems the bridge module has
trouble, too. rmmod bridge, or just ip li set dev br0 down makes timer
go from crazy to normal, too.

Thanks for the hint, I think I can manage to find a working
configuration based on this. Of course, having no such problems in the
first place would be even better.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: Serious performance issues

[Tommi Virtanen]

Ian Pratt wrote:
> Just to be clear, you're not using the usb virtualization stuff, just
> accessing the device in domain 0?

During these tests, no domU was ever started.

> Does anyone else see this interrupt storm with USB devices active?
> Please can everyone check as I'll hold the 2.0.5 release (with USB
> enabled in the default config) until we understand the scale of the
> problem.  

Also, note that I can get the same problem with bridging:

lots of timer interrupts
ip li set dev br0 down --> timer gets 30/s
ip li set dev br0 up --> timer goes crazy


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

RE: Serious performance issues

[Ian Pratt]

> Ian Pratt wrote:
> > Have you got some interrupt line going crazy?
> 
> That seems to be the problem.
> 
> In native linux, I see:
> 
> 0	timer	1000/s
> LOC		1000/s
> 
> and in xen dom0 I see:
> 
> 21	uhci_hcd	35000/s
> 130	timer		45000/s
> 
> rmmodding uhci_hcd removes irq 21, but makes the timer
> speed up to 115000/s.
> 
> rmmod ohci13943, rmmod ehci_hcd makes the timer go
> 800000/s.

Just to be clear, you're not using the usb virtualization stuff, just
accessing the device in domain 0?

Does anyone else see this interrupt storm with USB devices active?
Please can everyone check as I'll hold the 2.0.5 release (with USB
enabled in the default config) until we understand the scale of the
problem.  

Thanks,
Ian



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_ide95&alloc_id\x14396&op=click

RE: Serious performance issues

[Ian Pratt]

> Also, note that I can get the same problem with bridging:
> 
> lots of timer interrupts
> ip li set dev br0 down --> timer gets 30/s
> ip li set dev br0 up --> timer goes crazy

I certainly don't see this on any of my systems.

Are you using the standard bridging scripts? I guess not since your
bridge name is different. The only time I've seen something like this is
when some of the bridge timeout parameters are set to 0 jiffies (setting
to 1 jiffy is OK).

Ian


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_ide95&alloc_id\x14396&op=click

Re: Serious performance issues

[Tommi Virtanen]

Ian Pratt wrote:
> Are you using the standard bridging scripts? I guess not since your
> bridge name is different. The only time I've seen something like this is
> when some of the bridge timeout parameters are set to 0 jiffies (setting
> to 1 jiffy is OK).

I'm not. I had a fully working UML setup and wanted to share that
between UML and Xen.

The 0 jiffie thing was a good hint. That was the problem with bridge.
I'm guessing I had that issue all along with my native linux setup.

Summary:

Problem #1: plugging in my Logitech USB trackball creates an uhci_hcd
and timer interrupt storm that degrades performance to about 10% of
normal. Unplug does not stop storm. Unknown cause, not fixed, workaround
is not to plug in any USB device except hubs.

Problem #2: bridge setfd/sethello 0 causes timer interrupt storm, but
does not degrade performance significantly. Workaround is to not set
such low numbers. Will talk to brctl/bridging maintainers to make it
more foolproof.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

RE: Serious performance issues

[Ian Pratt]

> Problem #1: plugging in my Logitech USB trackball creates an uhci_hcd
> and timer interrupt storm that degrades performance to about 10% of
> normal. Unplug does not stop storm. Unknown cause, not fixed, 
> workaround
> is not to plug in any USB device except hubs.

Could you try using some of the kernel profile stuff to figure out where
the time is being spent? 

> Problem #2: bridge setfd/sethello 0 causes timer interrupt storm, but
> does not degrade performance significantly. Workaround is to not set
> such low numbers. Will talk to brctl/bridging maintainers to make it
> more foolproof.

If you're starting a dialogue with the bridge folk, I'd be grateful if
you could bring up the issue of transferring IP addresses from
interfaces that are added to a bridge to the bridge.

Thanks,
Ian


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_ide95&alloc_id\x14396&op=click

Re: Serious performance issues

[Nuutti Kotivuori]

Ian Pratt wrote:
> If you're starting a dialogue with the bridge folk, I'd be grateful
> if you could bring up the issue of transferring IP addresses from
> interfaces that are added to a bridge to the bridge.

Transferring IP addresses is not enough - if there is a dhclient
running for eth0 on the host, there is no way to change it to be
running for br0 transparently. So all we get is a seemingly well
working system - until the dhclient renews the lease.

The only solution to these kind of issues globally that I'd see, would
be the possibility to have packets come "up" from the eth0 interface,
instead of the br0 interface - which would need to be somehow
configurable.

-- Naked



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: Serious performance issues

[Nuutti Kotivuori]

Tommi Virtanen wrote:
> Problem #2: bridge setfd/sethello 0 causes timer interrupt storm,
> but does not degrade performance significantly. Workaround is to not
> set such low numbers. Will talk to brctl/bridging maintainers to
> make it more foolproof.

This is really odd. We run a lot of bridges with such settings. Or, to
be specific, for us, 'setfd' is always 0, 'setageing' is either 0 or
300, depending if we want the bridge to act as a 'hub' or not. STP is
always off.

Under these circumstances, we have never seen a timer interrupt
storm. But perhaps it is particular to having STP on.

-- Naked




-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: Re: Serious performance issues

[B.G. Bruce]

Can you do a "lspci -v" as root and see if there is any correlation
between your network adapters and your usb devices?  (same interrupt) or
something.

Thanks,
B.



On Mon, 2005-03-07 at 15:19, Nuutti Kotivuori wrote:
> Tommi Virtanen wrote:
> > Problem #2: bridge setfd/sethello 0 causes timer interrupt storm,
> > but does not degrade performance significantly. Workaround is to not
> > set such low numbers. Will talk to brctl/bridging maintainers to
> > make it more foolproof.
> 
> This is really odd. We run a lot of bridges with such settings. Or, to
> be specific, for us, 'setfd' is always 0, 'setageing' is either 0 or
> 300, depending if we want the bridge to act as a 'hub' or not. STP is
> always off.
> 
> Under these circumstances, we have never seen a timer interrupt
> storm. But perhaps it is particular to having STP on.
> 
> -- Naked
> 
> 
> 
> 
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
> 


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click