From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Schinzel Subject: RE: Read Performance issue when Xen Hypervisor is activated Date: Tue, 27 Dec 2016 14:26:55 +0000 Message-ID: <64f352a7a8a14ec19da5d07374b48eb6@ip-projects.de> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0990948988090690994==" Return-path: Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cLsiR-0002UD-J0 for xen-devel@lists.xenproject.org; Tue, 27 Dec 2016 14:26:59 +0000 Received: from [10.5.0.53] (port=52514 helo=exch2.ipp-hex.lan) by mail-gateway.ip-projects.de with esmtps (TLSv1.2:AES256-SHA256:256) (Exim 4.82_1-5b7a7c0-XX) (envelope-from ) id 1cLsiO-0003YA-09 for xen-devel@lists.xenproject.org; Tue, 27 Dec 2016 15:26:56 +0100 Content-Language: de-DE List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" To: Xen-devel List-Id: xen-devel@lists.xenproject.org --===============0990948988090690994== Content-Language: de-DE Content-Type: multipart/alternative; boundary="_000_64f352a7a8a14ec19da5d07374b48eb6ipprojectsde_" --_000_64f352a7a8a14ec19da5d07374b48eb6ipprojectsde_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable We have searched in the last days more and more for the cause of this perfo= rmance issue. In cooperation with the datacenter, we change some hardware to check, if th= e problem already proceeds. We put the RAID Controller included all RAID Ar= rays to another Supermicro Mainboard: X10SLM-F with only one CPU. The resul= t was, we got 400 MB/s read Speed. So it seems there is an issue with the S= ervers Mainboard / CPU and the Xen Hypervisor but, we also change the Mainb= oard to an Supermicro X9DR3-F with the actual BIOS Version 3.2a - these als= o do not solved the problem with the performance. What we also have done: - Upgraded Hypervisor from default Debian 8 - 4.4.1 to 4.8. - Tested some kernel boot configurations With an non hypervisor Kernel, the system also uses the read Cache of the c= ontroller and after some read operations at the same file, it gets 1.2 G/s = back from the Cache. At Xen Hypervisor Kernel, it seems the system do not u= se any caching operations. I also tested a bit with hdparm: root@v7:~# hdparm -Tt /dev/sdb /dev/sdb: Timing cached reads: 14060 MB in 1.99 seconds =3D 7076.16 MB/sec Timing buffered disk reads: 304 MB in 3.01 seconds =3D 100.85 MB/sec This Performance is horrable. It is a RAID 10 with read/write cache and SSD= Caching functions. Does somebody know how Xen proceeds with such Caching Systems? Yours sincerely Michael Schinzel --_000_64f352a7a8a14ec19da5d07374b48eb6ipprojectsde_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

We have searched in the last days more and more for = the cause of this performance issue.

 

In cooperation with the datacenter, we change some h= ardware to check, if the problem already proceeds. We put the RAID Controll= er included all RAID Arrays to another Supermicro Mainboard: X10SLM-F with = only one CPU. The result was, we got 400 MB/s read Speed. So it seems there is an issue with the Servers Mainbo= ard / CPU and the Xen Hypervisor but, we also change the Mainboard to an Su= permicro X9DR3-F with the actual BIOS Version 3.2a – these also do no= t solved the problem with the performance.

 

What we also have done:

-     &= nbsp;    Upgraded Hypervisor from default Debian 8 – 4= .4.1 to 4.8.

-     &= nbsp;    Tested some kernel boot configurations

 

With an non hypervisor Kernel, the system also uses = the read Cache of the controller and after some read operations at the same= file, it gets 1.2 G/s back from the Cache. At Xen Hypervisor Kernel, it se= ems the system do not use any caching operations. I also tested a bit with hdparm:

 

root@v7:~# hdparm -Tt /dev/sdb

 

/dev/sdb:

Timing cached reads:   14060 MB in  1= .99 seconds =3D 7076.16 MB/sec

Timing buffered disk reads: 304 MB in  3.01 sec= onds =3D 100.85 MB/sec

 

This Performance is horrable. It is a RAID 10 with r= ead/write cache and SSD Caching functions.

 

Does somebody know how Xen proceeds with such Cachin= g Systems?

 

 <= /o:p>

Yours sincer= ely

 <= /o:p>

Michael Schi= nzel

 

 

--_000_64f352a7a8a14ec19da5d07374b48eb6ipprojectsde_-- --===============0990948988090690994== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KWGVuLWRldmVs IG1haWxpbmcgbGlzdApYZW4tZGV2ZWxAbGlzdHMueGVuLm9yZwpodHRwczovL2xpc3RzLnhlbi5v cmcveGVuLWRldmVsCg== --===============0990948988090690994==--