4.1.2 very slow without upstream patches, but fast with them, also 4.2 very slow

4.1.2 very slow without upstream patches, but fast with them, also 4.2 very slow

[Peter Maloney]

my AMD FX-8150 system with vanilla source code is super slow, both the
dom0 and domUs. However, after I merge the upstream patches I found in
the openSUSE rpm, it runs normally.

I tried 4.2-unstable and it was the same. There was no rc1 when I tested
it about 1.5 weeks ago. And 4.2 has the same horrible performance, and
obviously those patches won't work any more since the 4.2 code looks
completely reorganized, so I'm stuck with 4.1.2

Here is the rpm I was using at the time:
http://download.opensuse.org/update/12.1/src/xen-4.1.2_16-1.7.1.src.rpm

To see the list of the patches and what order to apply them, see the
spec file.

Please make sure this performance issue is fixed for the 4.2 release.
And I would be happy to test whatever files you send me.

Re: 4.1.2 very slow without upstream patches, but fast with them, also 4.2 very slow

[Andrew Cooper]

On 06/08/12 11:12, Peter Maloney wrote:
> my AMD FX-8150 system with vanilla source code is super slow, both the
> dom0 and domUs. However, after I merge the upstream patches I found in
> the openSUSE rpm, it runs normally.
>
> I tried 4.2-unstable and it was the same. There was no rc1 when I tested
> it about 1.5 weeks ago. And 4.2 has the same horrible performance, and
> obviously those patches won't work any more since the 4.2 code looks
> completely reorganized, so I'm stuck with 4.1.2
>
> Here is the rpm I was using at the time:
> http://download.opensuse.org/update/12.1/src/xen-4.1.2_16-1.7.1.src.rpm
>
> To see the list of the patches and what order to apply them, see the
> spec file.
>
> Please make sure this performance issue is fixed for the 4.2 release.
> And I would be happy to test whatever files you send me.
>

Without identifying which patch or patches make a difference for you,
there is very little we can do.  There are 406 patches in that spec
file.  Furthermore, from the file names, I would say that most of the
patches have been backported from unstable.

>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

-- 
Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
T: +44 (0)1223 225 900, http://www.citrix.com

Re: 4.1.2 very slow without upstream patches, but fast with them, also 4.2 very slow

[Jan Beulich]

>>> On 06.08.12 at 12:12, Peter Maloney <peter.maloney@brockmann-consult.de> wrote:
> my AMD FX-8150 system with vanilla source code is super slow, both the
> dom0 and domUs. However, after I merge the upstream patches I found in
> the openSUSE rpm, it runs normally.

I'd be very surprised if you really just took the upstream patches,
and the result was better than 4.2-rc1. After all, what upstream
means is that they were taken from -unstable.

> I tried 4.2-unstable and it was the same. There was no rc1 when I tested
> it about 1.5 weeks ago. And 4.2 has the same horrible performance, and
> obviously those patches won't work any more since the 4.2 code looks
> completely reorganized, so I'm stuck with 4.1.2

Obviously the upstream patches can't be applied to something
that already has all those changes. Other patches, of which we
unfortunately have quite a few, would be a different story.

> Here is the rpm I was using at the time:
> http://download.opensuse.org/update/12.1/src/xen-4.1.2_16-1.7.1.src.rpm 
> 
> To see the list of the patches and what order to apply them, see the
> spec file.

That still won't tell us which patches you did apply.

> Please make sure this performance issue is fixed for the 4.2 release.
> And I would be happy to test whatever files you send me.

The sort of report you're doing isn't that helpful. What would
help is if you could narrow down which patch(es) it is that
make things so much better. Giving 4.1.3-rc a try might also
be worthwhile, albeit I would hope we don't have a regression
in 4.2.0-rc compared to 4.1.3-rc...

Jan

Re: 4.1.2 very slow without upstream patches, but fast with them, also 4.2 very slow

[Malcolm Crossley]

On 06/08/12 11:12, Peter Maloney wrote:
> my AMD FX-8150 system with vanilla source code is super slow, both the
> dom0 and domUs. However, after I merge the upstream patches I found in
> the openSUSE rpm, it runs normally.
>
> I tried 4.2-unstable and it was the same. There was no rc1 when I tested
> it about 1.5 weeks ago. And 4.2 has the same horrible performance, and
> obviously those patches won't work any more since the 4.2 code looks
> completely reorganized, so I'm stuck with 4.1.2
>
> Here is the rpm I was using at the time:
> http://download.opensuse.org/update/12.1/src/xen-4.1.2_16-1.7.1.src.rpm
>
> To see the list of the patches and what order to apply them, see the
> spec file.
>
> Please make sure this performance issue is fixed for the 4.2 release.
> And I would be happy to test whatever files you send me.
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
I suspect you may need the following patch to improve your 4.1.2 
performance:

http://xenbits.xen.org/hg/xen-4.1-testing.hg/rev/435493696053

The cache flush on every C2 transition is very expensive and causes a 
large slow down.

4.1.3-rc3 already includes that patch so it would be worth testing that 
version.

Malcolm

Re: 4.1.2 very slow without upstream patches, but fast with them, also 4.2 very slow

[Konrad Rzeszutek Wilk]

On Mon, Aug 06, 2012 at 01:41:07PM +0100, Malcolm Crossley wrote:
> On 06/08/12 11:12, Peter Maloney wrote:
> >my AMD FX-8150 system with vanilla source code is super slow, both the
> >dom0 and domUs. However, after I merge the upstream patches I found in
> >the openSUSE rpm, it runs normally.
> >
> >I tried 4.2-unstable and it was the same. There was no rc1 when I tested
> >it about 1.5 weeks ago. And 4.2 has the same horrible performance, and
> >obviously those patches won't work any more since the 4.2 code looks
> >completely reorganized, so I'm stuck with 4.1.2
> >
> >Here is the rpm I was using at the time:
> >http://download.opensuse.org/update/12.1/src/xen-4.1.2_16-1.7.1.src.rpm
> >
> >To see the list of the patches and what order to apply them, see the
> >spec file.
> >
> >Please make sure this performance issue is fixed for the 4.2 release.
> >And I would be happy to test whatever files you send me.
> >
> >
> >
> >_______________________________________________
> >Xen-devel mailing list
> >Xen-devel@lists.xen.org
> >http://lists.xen.org/xen-devel
> I suspect you may need the following patch to improve your 4.1.2
> performance:
> 
> http://xenbits.xen.org/hg/xen-4.1-testing.hg/rev/435493696053
> 
> The cache flush on every C2 transition is very expensive and causes
> a large slow down.
> 
> 4.1.3-rc3 already includes that patch so it would be worth testing
> that version.

MA Young, could this be back-ported in the F17 and F16. I belive
Micahel Petullo setup a bug for that?

> 
> Malcolm
> 
> 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: 4.1.2 very slow without upstream patches, but fast with them, also 4.2 very slow

[Peter Maloney]

> That still won't tell us which patches you did apply.
I applied no patches and tested, and the result was slow. And then
applied all patches, and it was fast. I didn't try figuring out which
one it was.


So I guess I'll try:
- the latest unstable 4.2
- the 4.1.3-rc (Which includes the patch Malcolm suggested)
- and my rpm source with half patches, 3/4 of them, etc. binary search
style to see which patch(es) changed the performance. But this means I
won't be able to narrow it down to a single patch, but only the point in
the long list where the most dramatic change happens, possibly depending
on many previous patches.

Thanks so far, guys.


On 08/06/2012 12:31 PM, Jan Beulich wrote:
>>>> On 06.08.12 at 12:12, Peter Maloney <peter.maloney@brockmann-consult.de> wrote:
>> my AMD FX-8150 system with vanilla source code is super slow, both the
>> dom0 and domUs. However, after I merge the upstream patches I found in
>> the openSUSE rpm, it runs normally.
> I'd be very surprised if you really just took the upstream patches,
> and the result was better than 4.2-rc1. After all, what upstream
> means is that they were taken from -unstable.
>
>> I tried 4.2-unstable and it was the same. There was no rc1 when I tested
>> it about 1.5 weeks ago. And 4.2 has the same horrible performance, and
>> obviously those patches won't work any more since the 4.2 code looks
>> completely reorganized, so I'm stuck with 4.1.2
> Obviously the upstream patches can't be applied to something
> that already has all those changes. Other patches, of which we
> unfortunately have quite a few, would be a different story.
>
>> Here is the rpm I was using at the time:
>> http://download.opensuse.org/update/12.1/src/xen-4.1.2_16-1.7.1.src.rpm 
>>
>> To see the list of the patches and what order to apply them, see the
>> spec file.
> That still won't tell us which patches you did apply.
>
>> Please make sure this performance issue is fixed for the 4.2 release.
>> And I would be happy to test whatever files you send me.
> The sort of report you're doing isn't that helpful. What would
> help is if you could narrow down which patch(es) it is that
> make things so much better. Giving 4.1.3-rc a try might also
> be worthwhile, albeit I would hope we don't have a regression
> in 4.2.0-rc compared to 4.1.3-rc...
>
> Jan
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel


-- 

--------------------------------------------
Peter Maloney
Brockmann Consult
Max-Planck-Str. 2
21502 Geesthacht
Germany
Tel: +49 4152 889 300
Fax: +49 4152 889 333
E-mail: peter.maloney@brockmann-consult.de
Internet: http://www.brockmann-consult.de
--------------------------------------------

Re: 4.1.2 very slow without upstream patches, but fast with them, also 4.2 very slow

[M A Young]

On Mon, 6 Aug 2012, Konrad Rzeszutek Wilk wrote:
> On Mon, Aug 06, 2012 at 01:41:07PM +0100, Malcolm Crossley wrote:

>> I suspect you may need the following patch to improve your 4.1.2
>> performance:
>>
>> http://xenbits.xen.org/hg/xen-4.1-testing.hg/rev/435493696053
>>
>> The cache flush on every C2 transition is very expensive and causes
>> a large slow down.
>>
>> 4.1.3-rc3 already includes that patch so it would be worth testing
>> that version.
>
> MA Young, could this be back-ported in the F17 and F16. I belive
> Micahel Petullo setup a bug for that?

I don't recall I seeing a bug for this but the patch is in 
xen-4.1.2-25.fc18 and xen-4.1.2-25.fc17 (which is building now).

 	Michael Young

Re: 4.1.2 very slow without upstream patches, but fast with them, also 4.2 very slow

[W. Michael Petullo]

>>> I suspect you may need the following patch to improve your 4.1.2
>>> performance:
>>>
>>> http://xenbits.xen.org/hg/xen-4.1-testing.hg/rev/435493696053
>>>
>>> The cache flush on every C2 transition is very expensive and causes
>>> a large slow down.

>>> 4.1.3-rc3 already includes that patch so it would be worth testing
>>> that version.

>> MA Young, could this be back-ported in the F17 and F16. I belive
>> Micahel Petullo setup a bug for that?
 
> I don't recall I seeing a bug for this but the patch is in
> xen-4.1.2-25.fc18 and xen-4.1.2-25.fc17 (which is building now).

The performance-related bug I filed is at:

https://bugzilla.redhat.com/show_bug.cgi?id=841330

-- 
Mike

:wq

Re: 4.1.2 very slow without upstream patches, but fast with them, also 4.2 very slow

[Peter Maloney]

So... did my 4.2-unstable test, using a fresh pull from yesterday; dom0
is normal fast (unlike previous tests), and domU is ultra slow, but
actually boots, and graphics card passthrough works without any patches,
and so does the USB keyboard, but USB mouse passthrough doesn't work.


On 08/07/2012 09:25 AM, Peter Maloney wrote:
>> That still won't tell us which patches you did apply.
> I applied no patches and tested, and the result was slow. And then
> applied all patches, and it was fast. I didn't try figuring out which
> one it was.
>
>
> So I guess I'll try:
> - the latest unstable 4.2
> - the 4.1.3-rc (Which includes the patch Malcolm suggested)
> - and my rpm source with half patches, 3/4 of them, etc. binary search
> style to see which patch(es) changed the performance. But this means I
> won't be able to narrow it down to a single patch, but only the point in
> the long list where the most dramatic change happens, possibly depending
> on many previous patches.
>
> Thanks so far, guys.
>
>
> On 08/06/2012 12:31 PM, Jan Beulich wrote:
>>>>> On 06.08.12 at 12:12, Peter Maloney <peter.maloney@brockmann-consult.de> wrote:
>>> my AMD FX-8150 system with vanilla source code is super slow, both the
>>> dom0 and domUs. However, after I merge the upstream patches I found in
>>> the openSUSE rpm, it runs normally.
>> I'd be very surprised if you really just took the upstream patches,
>> and the result was better than 4.2-rc1. After all, what upstream
>> means is that they were taken from -unstable.
>>
>>> I tried 4.2-unstable and it was the same. There was no rc1 when I tested
>>> it about 1.5 weeks ago. And 4.2 has the same horrible performance, and
>>> obviously those patches won't work any more since the 4.2 code looks
>>> completely reorganized, so I'm stuck with 4.1.2
>> Obviously the upstream patches can't be applied to something
>> that already has all those changes. Other patches, of which we
>> unfortunately have quite a few, would be a different story.
>>
>>> Here is the rpm I was using at the time:
>>> http://download.opensuse.org/update/12.1/src/xen-4.1.2_16-1.7.1.src.rpm 
>>>
>>> To see the list of the patches and what order to apply them, see the
>>> spec file.
>> That still won't tell us which patches you did apply.
>>
>>> Please make sure this performance issue is fixed for the 4.2 release.
>>> And I would be happy to test whatever files you send me.
>> The sort of report you're doing isn't that helpful. What would
>> help is if you could narrow down which patch(es) it is that
>> make things so much better. Giving 4.1.3-rc a try might also
>> be worthwhile, albeit I would hope we don't have a regression
>> in 4.2.0-rc compared to 4.1.3-rc...
>>
>> Jan
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel
>

Re: 4.1.2 very slow without upstream patches, but fast with them, also 4.2 very slow

[Peter Maloney]

I also tested 4.1.3, which is fast, and both USB and graphics
passthrough work, but "xl create" gave this message the first time I
started the vm (but not the second):

libxl: error: libxl_pci.c:750:libxl_device_pci_reset The kernel doesn't
support reset from sysfs for PCI device 0000:00:12.0


0000:00:12.0 is a USB device, which works in the VM.

peter:/opt # lspci -v | grep 00:12.0
00:12.0 USB Controller: ATI Technologies Inc SB7x0/SB8x0/SB9x0 USB OHCI0
Controller (prog-if 10 [OHCI])


On 08/13/2012 08:54 PM, Peter Maloney wrote:
> So... did my 4.2-unstable test, using a fresh pull from yesterday; dom0
> is normal fast (unlike previous tests), and domU is ultra slow, but
> actually boots, and graphics card passthrough works without any patches,
> and so does the USB keyboard, but USB mouse passthrough doesn't work.
>
>
> On 08/07/2012 09:25 AM, Peter Maloney wrote:
>>> That still won't tell us which patches you did apply.
>> I applied no patches and tested, and the result was slow. And then
>> applied all patches, and it was fast. I didn't try figuring out which
>> one it was.
>>
>>
>> So I guess I'll try:
>> - the latest unstable 4.2
>> - the 4.1.3-rc (Which includes the patch Malcolm suggested)
>> - and my rpm source with half patches, 3/4 of them, etc. binary search
>> style to see which patch(es) changed the performance. But this means I
>> won't be able to narrow it down to a single patch, but only the point in
>> the long list where the most dramatic change happens, possibly depending
>> on many previous patches.
>>
>> Thanks so far, guys.
>>
>>
>> On 08/06/2012 12:31 PM, Jan Beulich wrote:
>>>>>> On 06.08.12 at 12:12, Peter Maloney <peter.maloney@brockmann-consult.de> wrote:
>>>> my AMD FX-8150 system with vanilla source code is super slow, both the
>>>> dom0 and domUs. However, after I merge the upstream patches I found in
>>>> the openSUSE rpm, it runs normally.
>>> I'd be very surprised if you really just took the upstream patches,
>>> and the result was better than 4.2-rc1. After all, what upstream
>>> means is that they were taken from -unstable.
>>>
>>>> I tried 4.2-unstable and it was the same. There was no rc1 when I tested
>>>> it about 1.5 weeks ago. And 4.2 has the same horrible performance, and
>>>> obviously those patches won't work any more since the 4.2 code looks
>>>> completely reorganized, so I'm stuck with 4.1.2
>>> Obviously the upstream patches can't be applied to something
>>> that already has all those changes. Other patches, of which we
>>> unfortunately have quite a few, would be a different story.
>>>
>>>> Here is the rpm I was using at the time:
>>>> http://download.opensuse.org/update/12.1/src/xen-4.1.2_16-1.7.1.src.rpm 
>>>>
>>>> To see the list of the patches and what order to apply them, see the
>>>> spec file.
>>> That still won't tell us which patches you did apply.
>>>
>>>> Please make sure this performance issue is fixed for the 4.2 release.
>>>> And I would be happy to test whatever files you send me.
>>> The sort of report you're doing isn't that helpful. What would
>>> help is if you could narrow down which patch(es) it is that
>>> make things so much better. Giving 4.1.3-rc a try might also
>>> be worthwhile, albeit I would hope we don't have a regression
>>> in 4.2.0-rc compared to 4.1.3-rc...
>>>
>>> Jan
>>>
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xen.org
>>> http://lists.xen.org/xen-devel
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: 4.1.2 very slow without upstream patches, but fast with them, also 4.2 very slow

[Peter Maloney]

I ran some new tests... 4.1.2 with different patches, and
4.3-unstable.Some details are below.

At some point in the future, I will try some builds between 4.1 and 4.2
(but at the moment am not sure how with mercurial or what options I have).



4.1.2

short version: dom0 works fine; domu ran only in a few builds and works fine

long version:
I tested 4.1.2 again, with a few selections of patches (first n patches
where n was; 0, 23, 46, 93, 186, 279; there are 373 patches in total).
All of them ran fast in dom0, unlike when I first started this mailing
list thread, and the builds that would run my windows domu ran it fast.
So probably there was something strange with the kernel I had before,
which was probably 3.4.4; now I'm using something like 3.5.x (cmason's
linux-btrfs repository, for-linus branch)



4.3-unstable

short version: dom0 works fine; domu always runs terribly slow (which
leads me to wanting to test what changed between 4.1 and 4.2)


long version:
I pulled the latest source, built it, and dom0 is fast just like with
4.2, but windows hvm domu is still terribly slow, and (with 7 vcpus), it
consumes between 500-650% while booting and a few minutes afterwards.
With 4 CPUs, I would expect between 350-550% from observations with 4.2
but didn't test other cpu counts with 4.3. (another side note,  with
4.1.3 which is normally fast, it will run slow like 4.2 and 4.3 if I set
cpus="2,4,6,8" instead of cpus=4)

xentop - 11:32:44   Xen 4.3-unstable
2 domains: 2 running, 0 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown
Mem: 16757972k total, 16739788k used, 18184k free    CPUs: 8 @ 4499MHz
      NAME  STATE   CPU(sec) CPU(%)     MEM(k) MEM(%)  MAXMEM(k)
MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS   VBD_OO   VBD_RD   VBD_WR 
VBD_RSECT  VBD_WSECT SSID
  Domain-0 -----r        784   27.2   12314624   73.5   12582912     
75.1     8    0        0        0    0        0        0       
0          0          0    0
windowsxp2 -----r       3244  637.1    4197220   25.0    4198400     
25.1     7    1      344       56    2        0    14381     6054    
651283     122280    0

And then after idling for 10 minutes, it is under 200%

xentop - 11:35:29   Xen 4.3-unstable
2 domains: 1 running, 1 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown
Mem: 16757972k total, 16739788k used, 18184k free    CPUs: 8 @ 4499MHz
      NAME  STATE   CPU(sec) CPU(%)     MEM(k) MEM(%)  MAXMEM(k)
MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS   VBD_OO   VBD_RD   VBD_WR 
VBD_RSECT  VBD_WSECT SSID
  Domain-0 -----r        839   42.4   12314624   73.5   12582912     
75.1     8    0        0        0    0        0        0       
0          0          0    0
windowsxp2 --b---       3583  114.1    4197220   25.0    4198400     
25.1     7    1      426       66    2        0    14408     7501    
651853     180372    0


And then when it is in use (just loading a youtube page), it is up high
again.

xentop - 11:37:17   Xen 4.3-unstable
2 domains: 2 running, 0 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown
Mem: 16757972k total, 16739788k used, 18184k free    CPUs: 8 @ 4499MHz
      NAME  STATE   CPU(sec) CPU(%)     MEM(k) MEM(%)  MAXMEM(k)
MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS   VBD_OO   VBD_RD   VBD_WR 
VBD_RSECT  VBD_WSECT SSID
  Domain-0 -----r        875   37.8   12314624   73.5   12582912     
75.1     8    0        0        0    0        0        0       
0          0          0    0
windowsxp2 -----r       3945  529.7    4197220   25.0    4198400     
25.1     7    1     4885      201    2        0    17096     8168    
788573     198458    0


And also if I shut down the vm while it is at 600% cpu, it takes
something like 10-15 minutes to shut down.

and CPU temperature is only 45 degrees during the high cpu usage, and
26-32 degrees when it's 0%, so whatever CPU waste it's doing is not
generating much heat. With 4.1.3, while a game is open, it reports 200%
cpu, and the temperature is around 50 degrees. I have a huge CPU cooler;
it's overclocked; and normally it runs about 55-70 degrees using 8 cores
depending on the task.

I tested with apic=0 and apic=1 (apic=1 will run windows very slow in
4.1.x, so I have been using apic=0 normally)







On 08/13/2012 10:59 PM, Peter Maloney wrote:
> I also tested 4.1.3, which is fast, and both USB and graphics
> passthrough work, but "xl create" gave this message the first time I
> started the vm (but not the second):
>
> libxl: error: libxl_pci.c:750:libxl_device_pci_reset The kernel doesn't
> support reset from sysfs for PCI device 0000:00:12.0
>
>
> 0000:00:12.0 is a USB device, which works in the VM.
>
> peter:/opt # lspci -v | grep 00:12.0
> 00:12.0 USB Controller: ATI Technologies Inc SB7x0/SB8x0/SB9x0 USB OHCI0
> Controller (prog-if 10 [OHCI])
>
>
> On 08/13/2012 08:54 PM, Peter Maloney wrote:
>> So... did my 4.2-unstable test, using a fresh pull from yesterday; dom0
>> is normal fast (unlike previous tests), and domU is ultra slow, but
>> actually boots, and graphics card passthrough works without any patches,
>> and so does the USB keyboard, but USB mouse passthrough doesn't work.
>>
>>
>> On 08/07/2012 09:25 AM, Peter Maloney wrote:
>>>> That still won't tell us which patches you did apply.
>>> I applied no patches and tested, and the result was slow. And then
>>> applied all patches, and it was fast. I didn't try figuring out which
>>> one it was.
>>>
>>>
>>> So I guess I'll try:
>>> - the latest unstable 4.2
>>> - the 4.1.3-rc (Which includes the patch Malcolm suggested)
>>> - and my rpm source with half patches, 3/4 of them, etc. binary search
>>> style to see which patch(es) changed the performance. But this means I
>>> won't be able to narrow it down to a single patch, but only the point in
>>> the long list where the most dramatic change happens, possibly depending
>>> on many previous patches.
>>>
>>> Thanks so far, guys.
>>>
>>>
>>> On 08/06/2012 12:31 PM, Jan Beulich wrote:
>>>>>>> On 06.08.12 at 12:12, Peter Maloney <peter.maloney@brockmann-consult.de> wrote:
>>>>> my AMD FX-8150 system with vanilla source code is super slow, both the
>>>>> dom0 and domUs. However, after I merge the upstream patches I found in
>>>>> the openSUSE rpm, it runs normally.
>>>> I'd be very surprised if you really just took the upstream patches,
>>>> and the result was better than 4.2-rc1. After all, what upstream
>>>> means is that they were taken from -unstable.
>>>>
>>>>> I tried 4.2-unstable and it was the same. There was no rc1 when I tested
>>>>> it about 1.5 weeks ago. And 4.2 has the same horrible performance, and
>>>>> obviously those patches won't work any more since the 4.2 code looks
>>>>> completely reorganized, so I'm stuck with 4.1.2
>>>> Obviously the upstream patches can't be applied to something
>>>> that already has all those changes. Other patches, of which we
>>>> unfortunately have quite a few, would be a different story.
>>>>
>>>>> Here is the rpm I was using at the time:
>>>>> http://download.opensuse.org/update/12.1/src/xen-4.1.2_16-1.7.1.src.rpm 
>>>>>
>>>>> To see the list of the patches and what order to apply them, see the
>>>>> spec file.
>>>> That still won't tell us which patches you did apply.
>>>>
>>>>> Please make sure this performance issue is fixed for the 4.2 release.
>>>>> And I would be happy to test whatever files you send me.
>>>> The sort of report you're doing isn't that helpful. What would
>>>> help is if you could narrow down which patch(es) it is that
>>>> make things so much better. Giving 4.1.3-rc a try might also
>>>> be worthwhile, albeit I would hope we don't have a regression
>>>> in 4.2.0-rc compared to 4.1.3-rc...
>>>>
>>>> Jan
>>>>
>>>>
>>>> _______________________________________________
>>>> Xen-devel mailing list
>>>> Xen-devel@lists.xen.org
>>>> http://lists.xen.org/xen-devel
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: 4.1.2 very slow without upstream patches, but fast with them, also 4.2 very slow

[Peter Maloney]

[-- Attachment #1.1: Type: text/plain, Size: 8841 bytes --]

I also tested:
modprobe xen-acpi-processor

as suggeted by pasik on freenode. This didn't help (tested with xen
4.3-unstable)

And then also with 4.3-unstable, I tested 64 bit *windows8* preview,
*which seemed to run fast. *

So I guess this issue is specific to windows xp, or 32 bit domu in a 64
bit machine.


On 10/03/2012 07:19 PM, Peter Maloney wrote:
> I ran some new tests... 4.1.2 with different patches, and
> 4.3-unstable.Some details are below.
>
> At some point in the future, I will try some builds between 4.1 and 4.2
> (but at the moment am not sure how with mercurial or what options I have).
>
>
>
> 4.1.2
>
> short version: dom0 works fine; domu ran only in a few builds and works fine
>
> long version:
> I tested 4.1.2 again, with a few selections of patches (first n patches
> where n was; 0, 23, 46, 93, 186, 279; there are 373 patches in total).
> All of them ran fast in dom0, unlike when I first started this mailing
> list thread, and the builds that would run my windows domu ran it fast.
> So probably there was something strange with the kernel I had before,
> which was probably 3.4.4; now I'm using something like 3.5.x (cmason's
> linux-btrfs repository, for-linus branch)
>
>
>
> 4.3-unstable
>
> short version: dom0 works fine; domu always runs terribly slow (which
> leads me to wanting to test what changed between 4.1 and 4.2)
>
>
> long version:
> I pulled the latest source, built it, and dom0 is fast just like with
> 4.2, but windows hvm domu is still terribly slow, and (with 7 vcpus), it
> consumes between 500-650% while booting and a few minutes afterwards.
> With 4 CPUs, I would expect between 350-550% from observations with 4.2
> but didn't test other cpu counts with 4.3. (another side note,  with
> 4.1.3 which is normally fast, it will run slow like 4.2 and 4.3 if I set
> cpus="2,4,6,8" instead of cpus=4)
>
> xentop - 11:32:44   Xen 4.3-unstable
> 2 domains: 2 running, 0 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown
> Mem: 16757972k total, 16739788k used, 18184k free    CPUs: 8 @ 4499MHz
>       NAME  STATE   CPU(sec) CPU(%)     MEM(k) MEM(%)  MAXMEM(k)
> MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS   VBD_OO   VBD_RD   VBD_WR 
> VBD_RSECT  VBD_WSECT SSID
>   Domain-0 -----r        784   27.2   12314624   73.5   12582912     
> 75.1     8    0        0        0    0        0        0       
> 0          0          0    0
> windowsxp2 -----r       3244  637.1    4197220   25.0    4198400     
> 25.1     7    1      344       56    2        0    14381     6054    
> 651283     122280    0
>
> And then after idling for 10 minutes, it is under 200%
>
> xentop - 11:35:29   Xen 4.3-unstable
> 2 domains: 1 running, 1 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown
> Mem: 16757972k total, 16739788k used, 18184k free    CPUs: 8 @ 4499MHz
>       NAME  STATE   CPU(sec) CPU(%)     MEM(k) MEM(%)  MAXMEM(k)
> MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS   VBD_OO   VBD_RD   VBD_WR 
> VBD_RSECT  VBD_WSECT SSID
>   Domain-0 -----r        839   42.4   12314624   73.5   12582912     
> 75.1     8    0        0        0    0        0        0       
> 0          0          0    0
> windowsxp2 --b---       3583  114.1    4197220   25.0    4198400     
> 25.1     7    1      426       66    2        0    14408     7501    
> 651853     180372    0
>
>
> And then when it is in use (just loading a youtube page), it is up high
> again.
>
> xentop - 11:37:17   Xen 4.3-unstable
> 2 domains: 2 running, 0 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown
> Mem: 16757972k total, 16739788k used, 18184k free    CPUs: 8 @ 4499MHz
>       NAME  STATE   CPU(sec) CPU(%)     MEM(k) MEM(%)  MAXMEM(k)
> MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS   VBD_OO   VBD_RD   VBD_WR 
> VBD_RSECT  VBD_WSECT SSID
>   Domain-0 -----r        875   37.8   12314624   73.5   12582912     
> 75.1     8    0        0        0    0        0        0       
> 0          0          0    0
> windowsxp2 -----r       3945  529.7    4197220   25.0    4198400     
> 25.1     7    1     4885      201    2        0    17096     8168    
> 788573     198458    0
>
>
> And also if I shut down the vm while it is at 600% cpu, it takes
> something like 10-15 minutes to shut down.
>
> and CPU temperature is only 45 degrees during the high cpu usage, and
> 26-32 degrees when it's 0%, so whatever CPU waste it's doing is not
> generating much heat. With 4.1.3, while a game is open, it reports 200%
> cpu, and the temperature is around 50 degrees. I have a huge CPU cooler;
> it's overclocked; and normally it runs about 55-70 degrees using 8 cores
> depending on the task.
>
> I tested with apic=0 and apic=1 (apic=1 will run windows very slow in
> 4.1.x, so I have been using apic=0 normally)
>
>
>
>
>
>
>
> On 08/13/2012 10:59 PM, Peter Maloney wrote:
>> I also tested 4.1.3, which is fast, and both USB and graphics
>> passthrough work, but "xl create" gave this message the first time I
>> started the vm (but not the second):
>>
>> libxl: error: libxl_pci.c:750:libxl_device_pci_reset The kernel doesn't
>> support reset from sysfs for PCI device 0000:00:12.0
>>
>>
>> 0000:00:12.0 is a USB device, which works in the VM.
>>
>> peter:/opt # lspci -v | grep 00:12.0
>> 00:12.0 USB Controller: ATI Technologies Inc SB7x0/SB8x0/SB9x0 USB OHCI0
>> Controller (prog-if 10 [OHCI])
>>
>>
>> On 08/13/2012 08:54 PM, Peter Maloney wrote:
>>> So... did my 4.2-unstable test, using a fresh pull from yesterday; dom0
>>> is normal fast (unlike previous tests), and domU is ultra slow, but
>>> actually boots, and graphics card passthrough works without any patches,
>>> and so does the USB keyboard, but USB mouse passthrough doesn't work.
>>>
>>>
>>> On 08/07/2012 09:25 AM, Peter Maloney wrote:
>>>>> That still won't tell us which patches you did apply.
>>>> I applied no patches and tested, and the result was slow. And then
>>>> applied all patches, and it was fast. I didn't try figuring out which
>>>> one it was.
>>>>
>>>>
>>>> So I guess I'll try:
>>>> - the latest unstable 4.2
>>>> - the 4.1.3-rc (Which includes the patch Malcolm suggested)
>>>> - and my rpm source with half patches, 3/4 of them, etc. binary search
>>>> style to see which patch(es) changed the performance. But this means I
>>>> won't be able to narrow it down to a single patch, but only the point in
>>>> the long list where the most dramatic change happens, possibly depending
>>>> on many previous patches.
>>>>
>>>> Thanks so far, guys.
>>>>
>>>>
>>>> On 08/06/2012 12:31 PM, Jan Beulich wrote:
>>>>>>>> On 06.08.12 at 12:12, Peter Maloney <peter.maloney@brockmann-consult.de> wrote:
>>>>>> my AMD FX-8150 system with vanilla source code is super slow, both the
>>>>>> dom0 and domUs. However, after I merge the upstream patches I found in
>>>>>> the openSUSE rpm, it runs normally.
>>>>> I'd be very surprised if you really just took the upstream patches,
>>>>> and the result was better than 4.2-rc1. After all, what upstream
>>>>> means is that they were taken from -unstable.
>>>>>
>>>>>> I tried 4.2-unstable and it was the same. There was no rc1 when I tested
>>>>>> it about 1.5 weeks ago. And 4.2 has the same horrible performance, and
>>>>>> obviously those patches won't work any more since the 4.2 code looks
>>>>>> completely reorganized, so I'm stuck with 4.1.2
>>>>> Obviously the upstream patches can't be applied to something
>>>>> that already has all those changes. Other patches, of which we
>>>>> unfortunately have quite a few, would be a different story.
>>>>>
>>>>>> Here is the rpm I was using at the time:
>>>>>> http://download.opensuse.org/update/12.1/src/xen-4.1.2_16-1.7.1.src.rpm 
>>>>>>
>>>>>> To see the list of the patches and what order to apply them, see the
>>>>>> spec file.
>>>>> That still won't tell us which patches you did apply.
>>>>>
>>>>>> Please make sure this performance issue is fixed for the 4.2 release.
>>>>>> And I would be happy to test whatever files you send me.
>>>>> The sort of report you're doing isn't that helpful. What would
>>>>> help is if you could narrow down which patch(es) it is that
>>>>> make things so much better. Giving 4.1.3-rc a try might also
>>>>> be worthwhile, albeit I would hope we don't have a regression
>>>>> in 4.2.0-rc compared to 4.1.3-rc...
>>>>>
>>>>> Jan
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Xen-devel mailing list
>>>>> Xen-devel@lists.xen.org
>>>>> http://lists.xen.org/xen-devel
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xen.org
>>> http://lists.xen.org/xen-devel
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel


[-- Attachment #1.2: Type: text/html, Size: 10763 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.1.2 very slow without upstream patches, but fast with them, also 4.2 very slow

[Konrad Rzeszutek Wilk]

On Wed, Oct 03, 2012 at 10:05:24PM +0200, Peter Maloney wrote:
> I also tested:
> modprobe xen-acpi-processor

You didn't say what kind of CPU you have. Nor if you compiled
your kernel or if you used a distros' kernel.

One thing (just to eliminate this being a power management issue),
is to do this on the Linux command line:

xen-acpi-processor.off=1

It would also help if you provided the .config you are using. There
are some options you should not have one (like XEN_DEBUGFS.. something).

Re: 4.1.2 very slow without upstream patches, but fast with them, also 4.2 very slow

[Peter Maloney]

On 10/04/2012 04:25 PM, Konrad Rzeszutek Wilk wrote:
> On Wed, Oct 03, 2012 at 10:05:24PM +0200, Peter Maloney wrote:
>> I also tested:
>> modprobe xen-acpi-processor
> You didn't say what kind of CPU you have. Nor if you compiled
> your kernel or if you used a distros' kernel.
>
> One thing (just to eliminate this being a power management issue),
> is to do this on the Linux command line:
>
> xen-acpi-processor.off=1
>
> It would also help if you provided the .config you are using. There
> are some options you should not have one (like XEN_DEBUGFS.. something).
>
AMD FX-8150
990 FX chipset
I compiled the for-linus branch of cmason's linux-btrfs git repo (
git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git
for-linus )

Here's some hardware info: http://pastebin.com/XUZjmiVz
Here's the kernel config: http://pastebin.com/1GQbiFZE (only weird thing
I set was CONFIG_NR_CPUS=16 for no particular reason; default was 512 or
256)


3d stuff works fine (other than no es1370 sound device support) and fast
in windows8 preview also, without that 0fps 'warmup' / 'hicup' thing
happening (apparently as new textures are loading), which I wrote about
before in IRC but not in the mailing list.

At some point I plan on also testing:
- builds between 4.1 and xen 4.2 to find where the 32 bit windowsxp
starts going slow
- a 64 bit windows xp if I can find one... I don't have one at the moment
- a 32 bit windows 8 preview (I assume I can just download this like 64
bit win8 preview)

Re: 4.1.2 very slow without upstream patches, but fast with them, also 4.2 very slow

[Peter Maloney]

On 10/05/2012 10:19 PM, Peter Maloney wrote:
> On 10/04/2012 04:25 PM, Konrad Rzeszutek Wilk wrote:
>> On Wed, Oct 03, 2012 at 10:05:24PM +0200, Peter Maloney wrote:
>>> I also tested:
>>> modprobe xen-acpi-processor
>> You didn't say what kind of CPU you have. Nor if you compiled
>> your kernel or if you used a distros' kernel.
>>
>> One thing (just to eliminate this being a power management issue),
>> is to do this on the Linux command line:
>>
>> xen-acpi-processor.off=1
>>
>> It would also help if you provided the .config you are using. There
>> are some options you should not have one (like XEN_DEBUGFS.. something).
>>
> AMD FX-8150
> 990 FX chipset
> I compiled the for-linus branch of cmason's linux-btrfs git repo (
> git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git
> for-linus )
>
> Here's some hardware info: http://pastebin.com/XUZjmiVz
> Here's the kernel config: http://pastebin.com/1GQbiFZE (only weird thing
> I set was CONFIG_NR_CPUS=16 for no particular reason; default was 512 or
> 256)
>
>
> 3d stuff works fine (other than no es1370 sound device support) and fast
> in windows8 preview also, without that 0fps 'warmup' / 'hicup' thing
> happening (apparently as new textures are loading), which I wrote about
> before in IRC but not in the mailing list.
>
> At some point I plan on also testing:
> - builds between 4.1 and xen 4.2 to find where the 32 bit windowsxp
> starts going slow
> - a 64 bit windows xp if I can find one... I don't have one at the moment
> - a 32 bit windows 8 preview (I assume I can just download this like 64
> bit win8 preview)
32 bit Windows 8 preview is fast... so then it's only Windows xp that is
slow so far. And es1370 sound also works in 32 bit win8.