serious performance regression in Xen 4.2.4 and 4.2.5

serious performance regression in Xen 4.2.4 and 4.2.5

[Andreas Kinzler]

Hello Xen developers,

since we use Xen for our production systems, I run many tests on Xen 
(stability/performance). One test now uncovered a serious performance 
regression when updating from Xen 4.2.3 to 4.2.x (with x>=4). To 
reproduce run a domU (HVM) and compile a kernel for example ("time make 
-j3"). Below are my results - compilation times more than double!

Is it my mistake or otherwise how could such a bug be unnoticed? Really 
wondering.
If it is really a regression, will it be fixed since the 4.2 branch is 
declared unsupported?

Regards Andreas

Xen 4.2.3, 3.10.44-el7-pvops0-ak3, Xeon E3-1230, compile in HVM
     real    2m58.371s
     user    5m11.580s
     sys     0m12.736s

Xen 4.2.4, 3.10.44-el7-pvops0-ak3, Xeon E3-1230, compile in HVM
     real    6m44.697s    (!!!)
     user    12m4.988s
     sys     0m33.916s

Xen 4.2.5, 3.10.44-el7-pvops0-ak3, Xeon E3-1230, compile in HVM
     real    6m27.998s    (!!!)
     user    11m42.664s
     sys     0m32.700s

Re: serious performance regression in Xen 4.2.4 and 4.2.5

[Andrew Cooper]

On 19/02/15 11:08, Andreas Kinzler wrote:
> Hello Xen developers,
>
> since we use Xen for our production systems, I run many tests on Xen
> (stability/performance). One test now uncovered a serious performance
> regression when updating from Xen 4.2.3 to 4.2.x (with x>=4). To
> reproduce run a domU (HVM) and compile a kernel for example ("time
> make -j3"). Below are my results - compilation times more than double!
>
> Is it my mistake or otherwise how could such a bug be unnoticed?
> Really wondering.
> If it is really a regression, will it be fixed since the 4.2 branch is
> declared unsupported?
>
> Regards Andreas
>
> Xen 4.2.3, 3.10.44-el7-pvops0-ak3, Xeon E3-1230, compile in HVM
>     real    2m58.371s
>     user    5m11.580s
>     sys     0m12.736s
>
> Xen 4.2.4, 3.10.44-el7-pvops0-ak3, Xeon E3-1230, compile in HVM
>     real    6m44.697s    (!!!)
>     user    12m4.988s
>     sys     0m33.916s
>
> Xen 4.2.5, 3.10.44-el7-pvops0-ak3, Xeon E3-1230, compile in HVM
>     real    6m27.998s    (!!!)
>     user    11m42.664s
>     sys     0m32.700s
>
>

Is it perhaps
http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=f1e0df14412ccc6933a68eda66ac5b7d89186b62
?

There are a number of correctness fixes in that range which will
adversely affect performance.

~Andrew

Re: serious performance regression in Xen 4.2.4 and 4.2.5

[Malcolm Crossley]

On 19/02/15 11:08, Andreas Kinzler wrote:
> Hello Xen developers,
> 
> since we use Xen for our production systems, I run many tests on Xen
> (stability/performance). One test now uncovered a serious performance
> regression when updating from Xen 4.2.3 to 4.2.x (with x>=4). To
> reproduce run a domU (HVM) and compile a kernel for example ("time make
> -j3"). Below are my results - compilation times more than double!
> 
> Is it my mistake or otherwise how could such a bug be unnoticed? Really
> wondering.
> If it is really a regression, will it be fixed since the 4.2 branch is
> declared unsupported?
> 

I believe you are being hit with an issue with uncached mappings being
setup when RAM is relocated out of the guest MMIO hole:

http://lists.xen.org/archives/html/xen-devel/2015-02/msg02306.html

Only Intel system which do not support IOMMU snoop control are affected
which are typically desktop/workstation class processors.

You have the following workarounds:

You can configure the guest to have less than 3GB of RAM to workaround
the issue

or

Disable IOMMU support

or

Apply the patch I have linked to above.

Regards

Malcolm

> Regards Andreas
> 
> Xen 4.2.3, 3.10.44-el7-pvops0-ak3, Xeon E3-1230, compile in HVM
>     real    2m58.371s
>     user    5m11.580s
>     sys     0m12.736s
> 
> Xen 4.2.4, 3.10.44-el7-pvops0-ak3, Xeon E3-1230, compile in HVM
>     real    6m44.697s    (!!!)
>     user    12m4.988s
>     sys     0m33.916s
> 
> Xen 4.2.5, 3.10.44-el7-pvops0-ak3, Xeon E3-1230, compile in HVM
>     real    6m27.998s    (!!!)
>     user    11m42.664s
>     sys     0m32.700s
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: serious performance regression in Xen 4.2.4 and 4.2.5

[Jan Beulich]

>>> On 19.02.15 at 14:08, <malcolm.crossley@citrix.com> wrote:
> On 19/02/15 11:08, Andreas Kinzler wrote:
>> Hello Xen developers,
>> 
>> since we use Xen for our production systems, I run many tests on Xen
>> (stability/performance). One test now uncovered a serious performance
>> regression when updating from Xen 4.2.3 to 4.2.x (with x>=4). To
>> reproduce run a domU (HVM) and compile a kernel for example ("time make
>> -j3"). Below are my results - compilation times more than double!
>> 
>> Is it my mistake or otherwise how could such a bug be unnoticed? Really
>> wondering.
>> If it is really a regression, will it be fixed since the 4.2 branch is
>> declared unsupported?
>> 
> 
> I believe you are being hit with an issue with uncached mappings being
> setup when RAM is relocated out of the guest MMIO hole:
> 
> http://lists.xen.org/archives/html/xen-devel/2015-02/msg02306.html 
> 
> Only Intel system which do not support IOMMU snoop control are affected
> which are typically desktop/workstation class processors.
> 
> You have the following workarounds:
> 
> You can configure the guest to have less than 3GB of RAM to workaround
> the issue
> 
> or
> 
> Disable IOMMU support
> 
> or
> 
> Apply the patch I have linked to above.

I'm not really following: By default there shouldn't be any RAM
relocation - this ought to be needed only when there are passed
through devices with some pretty large BAR(s).

Jan

Re: serious performance regression in Xen 4.2.4 and 4.2.5

[Malcolm Crossley]

On 19/02/15 14:21, Jan Beulich wrote:
>>>> On 19.02.15 at 14:08, <malcolm.crossley@citrix.com> wrote:
>> On 19/02/15 11:08, Andreas Kinzler wrote:
>>> Hello Xen developers,
>>>
>>> since we use Xen for our production systems, I run many tests on Xen
>>> (stability/performance). One test now uncovered a serious performance
>>> regression when updating from Xen 4.2.3 to 4.2.x (with x>=4). To
>>> reproduce run a domU (HVM) and compile a kernel for example ("time make
>>> -j3"). Below are my results - compilation times more than double!
>>>
>>> Is it my mistake or otherwise how could such a bug be unnoticed? Really
>>> wondering.
>>> If it is really a regression, will it be fixed since the 4.2 branch is
>>> declared unsupported?
>>>
>>
>> I believe you are being hit with an issue with uncached mappings being
>> setup when RAM is relocated out of the guest MMIO hole:
>>
>> http://lists.xen.org/archives/html/xen-devel/2015-02/msg02306.html 
>>
>> Only Intel system which do not support IOMMU snoop control are affected
>> which are typically desktop/workstation class processors.
>>
>> You have the following workarounds:
>>
>> You can configure the guest to have less than 3GB of RAM to workaround
>> the issue
>>
>> or
>>
>> Disable IOMMU support
>>
>> or
>>
>> Apply the patch I have linked to above.
> 
> I'm not really following: By default there shouldn't be any RAM
> relocation - this ought to be needed only when there are passed
> through devices with some pretty large BAR(s).
> 
That's a good point, I had forgotten about that extra condition (we've
been testing GPU passthrough and so have always had large BAR's in the
guest).

Hopefully Andreas can give us more information on the host and guest
configuration.

> Jan
> 

Re: serious performance regression in Xen 4.2.4 and 4.2.5

[Andreas Kinzler]

On 19.02.2015 12:20, Andrew Cooper wrote:
>Is it perhaps
>http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=f1e0df14412ccc6933a68eda66ac5b7d89186b62
>There are a number of correctness fixes in that range which will
>adversely affect performance.

I now reverted:
   http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=f1e0df14412ccc6933a68eda66ac5b7d89186b62
   http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=644e6c5c7106d276fb750daf6e07de0034b2e76a
   http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=0fffcffeb594741d5027c8c0d05900b951f997b2
and benchmark times improve but are no longer reproducable (either 3:36 or 5:09 instead of 6:xx with untouched 4.2.4).

Xen 4.3.3/4.4.1/4.5.0 have performance nearly identical to 4.2.3.

System:
   Mainboard Supermicro X9SCM-F, BIOS 2.1
   Adaptec 5405, one Raid 0
   Xeon E3-1230
   16 GB ECC non-registered
   dom0 kernel is mostly vanilla 3.10.44
   VT-d is turned off in the BIOS

"linux compile" HVM has 2 vCPUs and 2000 MB RAM

Regards Andreas