From mboxrd@z Thu Jan 1 00:00:00 1970 From: Steven Haigh Subject: Re: 4.2.1: Poor write performance for DomU. Date: Tue, 12 Mar 2013 00:37:11 +1100 Message-ID: <513DDE07.2090607@crc.id.au> References: <51243089.8020307@crc.id.au> <512488AC.1020702@citrix.com> <51248E23.7060408@crc.id.au> <51249C11.3050800@crc.id.au> <5139A747.80703@crc.id.au> <20130308204916.GB22146@phenom.dumpdata.com> <513A669E.4060906@crc.id.au> <20130311133017.GJ23018@phenom.dumpdata.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============4170748680092524933==" Return-path: In-Reply-To: <20130311133017.GJ23018@phenom.dumpdata.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Konrad Rzeszutek Wilk Cc: "xen-devel@lists.xen.org" , =?ISO-8859-1?Q?Roger_Pau_Monn=E9?= List-Id: xen-devel@lists.xenproject.org This is a cryptographically signed message in MIME format. --===============4170748680092524933== Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha1; boundary="------------ms090306080802060506010307" This is a cryptographically signed message in MIME format. --------------ms090306080802060506010307 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable On 12/03/2013 12:30 AM, Konrad Rzeszutek Wilk wrote: > On Sat, Mar 09, 2013 at 09:30:54AM +1100, Steven Haigh wrote: >> On 9/03/2013 7:49 AM, Konrad Rzeszutek Wilk wrote: >>> On Fri, Mar 08, 2013 at 07:54:31PM +1100, Steven Haigh wrote: >>>> On 20/02/2013 8:49 PM, Steven Haigh wrote: >>>>> On 20/02/2013 7:49 PM, Steven Haigh wrote: >>>>>> On 20/02/2013 7:26 PM, Roger Pau Monn=E9 wrote: >>>>>>> On 20/02/13 03:10, Steven Haigh wrote: >>>>>>>> Hi guys, >>>>>>>> >>>>>>>> Firstly, please CC me in to any replies as I'm not a subscriber = these >>>>>>>> days. >>>>>>>> >>>>>>>> I've been trying to debug a problem with Xen 4.2.1 where I am un= able to >>>>>>>> achieve more than ~50Mb/sec sustained sequential write to a disk= =2E The >>>>>>>> DomU is configured as such: >>>>>>> >>>>>>> Since you mention 4.2.1 explicitly, is this a performance regress= ion >>>>>> >from previous versions? (4.2.0 or the 4.1 branch) >>>>>> >>>>>> This is actually a very good question. I've reinstalled my older >>>>>> packages of Xen 4.1.3 back on the system. Rebooting into the new >>>>>> hypervisor, then starting the single DomU again. Ran bonnie++ agai= n on >>>>>> the DomU: >>>>>> >>>>>> Still around 50Mb/sec - so this doesn't seem to be a regression, b= ut >>>>>> something else? >>>>> >>>>> I've actually done a bit of thinking about this... A recent thread = on >>>>> linux-raid kernel mailing list about Xen and DomU throughput made m= e >>>>> revisit my setup. I know I used to be able to saturate GigE both wa= ys >>>>> (send and receive) to the samba share served by this DomU. This wou= ld >>>>> mean I'd get at least 90-100Mbyte/sec. What exact config and kernel= /xen >>>>> versions this was as this point in time I cannot say. >>>>> >>>>> As such, I had a bit of a play and recreated my RAID6 with 64Kb chu= nk >>>>> size. This seemed to make rebuild/resync speeds way worse - so I >>>>> reverted to 128Kb chunk size. >>>>> >>>>> The benchmarks I am getting from the Dom0 is about what I'd expect = - but >>>>> I wouldn't expect to lose 130Mb/sec write speed to the phy:/ pass >>>>> through of the LV. >>>>> >>>>> From my known config where I could saturate the GigE connection, I= have >>>>> changed from kernel 2.6.32 (Jeremy's git repo) to the latest vanill= a >>>>> kernels - currently 3.7.9. >>>>> >>>>> My build of Xen 4.2.1 also has all of the recent security advisorie= s >>>>> patched as well. Although it is interesting to note that downgradin= g to >>>>> Xen 4.1.2 made no difference to write speeds. >>>>> >>>> >>>> Just wondering if there is any further news or tests that I might be= >>>> able to do on this? >>> >>> So usually the problem like this is to unpeel the layers and find out= >>> which of them is at fault. You have a stacked block system - LVM on >>> top of RAID6 on top of block devices. >>> >>> To figure out who is interferring with the speeds I would recommend >>> you fault one of the RAID6 disks (so take it out of the RAID6). Pass >>> it to the guest as a raw disk (/dev/sdX as /dev/xvd) and then >>> run 'fio'. Run 'fio' as well in dom0 on the /dev/sdX and check >>> whether the write performance is different. >>> >>> This is how I how do it: >>> >>> [/dev/xvdXXX] >>> rw=3Dwrite >>> direct=3D1 >>> size=3D4g >>> ioengine=3Dlibaio >>> iodepth=3D32 >>> >>> Then progress up the stack. Try sticking the disk back in RAID6 >>> and doing it on the RAID6. Then on the LVM and so on. >> >> I did try to peel it back a single layer at a time. My test was >> simply using the same XFS filesystem in the Dom0 instead of the >> DomU. > > Right, you are using a filesystem. That is another layer :-) > > And depending on what version of QEMU you have you might be using > QEMU as the block PV backend instead of the kernel one. There > were versions of QEMU that had highly inferior performance. > > Hence I was thinking of just using a raw disk to test that. > >> >> I tested the underlying LVM config by mounting >> /dev/vg_raid6/fileshare from within the Dom0 and running bonnie++ as >> a benchmark: > > > So still filesystem. Fio can do it on a block level. > > What does 'xenstore-ls' show you and 'losetup -a'? I am really > curious as to where that file you are providing to the guest as > disk is being handled via 'loop' or via 'QEMU'. > I've picked out what I believe is the most relevant from xenstore-ls=20 that belongs to the DomU in question: 1 =3D "" 51712 =3D "" domain =3D "zeus.vm" frontend =3D "/local/domain/1/device/vbd/51712" uuid =3D "3aa72be1-0e83-1ee2-a346-8ccef71e9d34" bootable =3D "1" dev =3D "xvda" state =3D "4" params =3D "/dev/RAID1/zeus.vm" mode =3D "w" online =3D "1" frontend-id =3D "1" type =3D "phy" physical-device =3D "fd:6" hotplug-status =3D "connected" feature-flush-cache =3D "1" feature-discard =3D "0" feature-barrier =3D "1" feature-persistent =3D "1" sectors =3D "135397376" info =3D "0" sector-size =3D "512" 51728 =3D "" domain =3D "zeus.vm" frontend =3D "/local/domain/1/device/vbd/51728" uuid =3D "28375672-321c-0e33-4549-d64ee4daadec" bootable =3D "0" dev =3D "xvdb" state =3D "4" params =3D "/dev/vg_raid6/fileshare" mode =3D "w" online =3D "1" frontend-id =3D "1" type =3D "phy" physical-device =3D "fd:5" hotplug-status =3D "connected" feature-flush-cache =3D "1" feature-discard =3D "0" feature-barrier =3D "1" feature-persistent =3D "1" sectors =3D "5368709120" info =3D "0" sector-size =3D "512" losetup -a returns nothing. --=20 Steven Haigh Email: netwiz@crc.id.au Web: https://www.crc.id.au Phone: (03) 9001 6090 - 0412 935 897 Fax: (03) 8338 0299 --------------ms090306080802060506010307 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIMdTCC BjQwggQcoAMCAQICAR4wDQYJKoZIhvcNAQEFBQAwfTELMAkGA1UEBhMCSUwxFjAUBgNVBAoT DVN0YXJ0Q29tIEx0ZC4xKzApBgNVBAsTIlNlY3VyZSBEaWdpdGFsIENlcnRpZmljYXRlIFNp Z25pbmcxKTAnBgNVBAMTIFN0YXJ0Q29tIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MB4XDTA3 MTAyNDIxMDE1NVoXDTE3MTAyNDIxMDE1NVowgYwxCzAJBgNVBAYTAklMMRYwFAYDVQQKEw1T dGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUgRGlnaXRhbCBDZXJ0aWZpY2F0ZSBTaWdu aW5nMTgwNgYDVQQDEy9TdGFydENvbSBDbGFzcyAxIFByaW1hcnkgSW50ZXJtZWRpYXRlIENs aWVudCBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMcJg8zOLdgasSmkLhOr lr6KMoOMpohBllVHrdRvEg/q6r8jR+EK75xCGhR8ToREoqe7zM9/UnC6TS2y9UKTpT1v7RSM zR0t6ndl0TWBuUr/UXBhPk+Kmy7bI4yW4urC+y7P3/1/X7U8ocb8VpH/Clt+4iq7nirMcNh6 qJR+xjOhV+VHzQMALuGYn5KZmc1NbJQYclsGkDxDz2UbFqE2+6vIZoL+jb9x4Pa5gNf1TwSD kOkikZB1xtB4ZqtXThaABSONdfmv/Z1pua3FYxnCFmdr/+N2JLKutIxMYqQOJebr/f/h5t95 m4JgrM3Y/w7YX9d7YAL9jvN4SydHsU6n65cCAwEAAaOCAa0wggGpMA8GA1UdEwEB/wQFMAMB Af8wDgYDVR0PAQH/BAQDAgEGMB0GA1UdDgQWBBRTcu2SnODaywFcfH6WNU7y1LhRgjAfBgNV HSMEGDAWgBROC+8apEBbpRdphzDKNGhD0EGu8jBmBggrBgEFBQcBAQRaMFgwJwYIKwYBBQUH MAGGG2h0dHA6Ly9vY3NwLnN0YXJ0c3NsLmNvbS9jYTAtBggrBgEFBQcwAoYhaHR0cDovL3d3 dy5zdGFydHNzbC5jb20vc2ZzY2EuY3J0MFsGA1UdHwRUMFIwJ6AloCOGIWh0dHA6Ly93d3cu c3RhcnRzc2wuY29tL3Nmc2NhLmNybDAnoCWgI4YhaHR0cDovL2NybC5zdGFydHNzbC5jb20v c2ZzY2EuY3JsMIGABgNVHSAEeTB3MHUGCysGAQQBgbU3AQIBMGYwLgYIKwYBBQUHAgEWImh0 dHA6Ly93d3cuc3RhcnRzc2wuY29tL3BvbGljeS5wZGYwNAYIKwYBBQUHAgEWKGh0dHA6Ly93 d3cuc3RhcnRzc2wuY29tL2ludGVybWVkaWF0ZS5wZGYwDQYJKoZIhvcNAQEFBQADggIBAAqD CH14qywGXLhjjF6uHLkjd02hcdh9hrw+VUsv+q1eeQWB21jWj3kJ96AUlPCoEGZ/ynJNScWy 6QMVQjbbMXltUfO4n4bGGdKo3awPWp61tjAFgraLJgDk+DsSvUD6EowjMTNx25GQgyYJ5RPI zKKR9tQW8gGK+2+RHxkUCTbYFnL6kl8Ch507rUdPPipJ9CgJFws3kDS3gOS5WFMxcjO5DwKf KSETEPrHh7p5shuuNktvsv6hxHTLhiMKX893gxdT3XLS9OKmCv87vkINQcNEcIIoFWbP9HOR z9v3vQwR4e3ksLc2JZOAFK+ssS5XMEoznzpihEP0PLc4dCBYjbvSD7kxgDwZ+Aj8Q9PkbvE9 sIPP7ON0fz095HdThKjiVJe6vofq+n6b1NBc8XdrQvBmunwxD5nvtTW4vtN6VY7mUCmxsCie uoBJ9OlqmsVWQvifIYf40dJPZkk9YgGTzWLpXDSfLSplbY2LL9C9U0ptvjcDjefLTvqSFc7t w1sEhF0n/qpA2r0GpvkLRDmcSwVyPvmjFBGqUp/pNy8ZuPGQmHwFi2/14+xeSUDG2bwnsYJQ G2EdJCB6luQ57GEnTA/yKZSTKI8dDQa8Sd3zfXb19mOgSF0bBdXbuKhEpuP9wirslFe6fQ1t 5j5R0xi72MZ8ikMu1RQZKCyDbMwazlHiMIIGOTCCBSGgAwIBAgIDBdMXMA0GCSqGSIb3DQEB BQUAMIGMMQswCQYDVQQGEwJJTDEWMBQGA1UEChMNU3RhcnRDb20gTHRkLjErMCkGA1UECxMi U2VjdXJlIERpZ2l0YWwgQ2VydGlmaWNhdGUgU2lnbmluZzE4MDYGA1UEAxMvU3RhcnRDb20g Q2xhc3MgMSBQcmltYXJ5IEludGVybWVkaWF0ZSBDbGllbnQgQ0EwHhcNMTMwMTI5MTIyMTM0 WhcNMTQwMTMwMjIzNjE4WjBXMRkwFwYDVQQNExA0MzM5TjFVWjZFS2ZBa0Y2MRkwFwYDVQQD DBBuZXR3aXpAY3JjLmlkLmF1MR8wHQYJKoZIhvcNAQkBFhBuZXR3aXpAY3JjLmlkLmF1MIIB IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAqz7eWA7Oi1ES6IIWwtRNtOa6cSj6GSio 72D26TkCPm/vYamqS/9pluIyMO2TbGjMyveGxWZB9bDt+MnuI6yJ3spD6br5TUXQSqsgp1/g NXicgNmUHESd/A7+POEdYKMyTtXjVEqmR8R6uPtD78MBhIKQVZdBYGFuRBhrjoFMOnWARmXD 47FG6eLH8jyeBi+RGEYEpBgTe1gFcoU1dCrjBqxAj47ALNgcHP4ffNX47DLvyhW2pMMaOYUh ga/UyQuPOpZcMAJdVI7dTIkbdipboEEE5pTdoIRI+F+XDLzO8bNkGLa1xLTWrbP048xxw3SL Aw/DmLYqQ+oC8vabAAqBIwIDAQABo4IC1jCCAtIwCQYDVR0TBAIwADALBgNVHQ8EBAMCBLAw HQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMB0GA1UdDgQWBBR/FxImYCufSdz0pG5w A8VPzNyaujAfBgNVHSMEGDAWgBRTcu2SnODaywFcfH6WNU7y1LhRgjAbBgNVHREEFDASgRBu ZXR3aXpAY3JjLmlkLmF1MIIBTAYDVR0gBIIBQzCCAT8wggE7BgsrBgEEAYG1NwECAzCCASow LgYIKwYBBQUHAgEWImh0dHA6Ly93d3cuc3RhcnRzc2wuY29tL3BvbGljeS5wZGYwgfcGCCsG AQUFBwICMIHqMCcWIFN0YXJ0Q29tIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MAMCAQEagb5U aGlzIGNlcnRpZmljYXRlIHdhcyBpc3N1ZWQgYWNjb3JkaW5nIHRvIHRoZSBDbGFzcyAxIFZh bGlkYXRpb24gcmVxdWlyZW1lbnRzIG9mIHRoZSBTdGFydENvbSBDQSBwb2xpY3ksIHJlbGlh bmNlIG9ubHkgZm9yIHRoZSBpbnRlbmRlZCBwdXJwb3NlIGluIGNvbXBsaWFuY2Ugb2YgdGhl IHJlbHlpbmcgcGFydHkgb2JsaWdhdGlvbnMuMDYGA1UdHwQvMC0wK6ApoCeGJWh0dHA6Ly9j cmwuc3RhcnRzc2wuY29tL2NydHUxLWNybC5jcmwwgY4GCCsGAQUFBwEBBIGBMH8wOQYIKwYB BQUHMAGGLWh0dHA6Ly9vY3NwLnN0YXJ0c3NsLmNvbS9zdWIvY2xhc3MxL2NsaWVudC9jYTBC BggrBgEFBQcwAoY2aHR0cDovL2FpYS5zdGFydHNzbC5jb20vY2VydHMvc3ViLmNsYXNzMS5j bGllbnQuY2EuY3J0MCMGA1UdEgQcMBqGGGh0dHA6Ly93d3cuc3RhcnRzc2wuY29tLzANBgkq hkiG9w0BAQUFAAOCAQEAwsdSJioQIH5mEXlMBkQBZN9qrqCvDn4e92NKIX2LMPhVH01o0weH MEgkFMOBypbcPrwnZD51IL+ZfPmuDrcFJR9SPk4W95p3wLakMUBmvDPJ3IC+pa97+fU/GYsh q4dth+VlAIWJarpPV5ee6hAXGQEZvC2jyy5ukcjlr/oQxegb4hufgkLhKjqaq5UDs2be5q1N CBewezVmAbb/VDNVogP1xacnHn4rhllskMfGAda2DPCCaSMaJ3LcC5JRyfMkpKe5UxpVmVeM vGP8hYGI9DlFjwpqRCmIvD/Wl6oWWJc0HaLl6k+U3dngqEHhunzh3H6b/3lWOmEKQuI3Piy+ oDGCA90wggPZAgEBMIGUMIGMMQswCQYDVQQGEwJJTDEWMBQGA1UEChMNU3RhcnRDb20gTHRk LjErMCkGA1UECxMiU2VjdXJlIERpZ2l0YWwgQ2VydGlmaWNhdGUgU2lnbmluZzE4MDYGA1UE AxMvU3RhcnRDb20gQ2xhc3MgMSBQcmltYXJ5IEludGVybWVkaWF0ZSBDbGllbnQgQ0ECAwXT FzAJBgUrDgMCGgUAoIICHTAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJ BTEPFw0xMzAzMTExMzM3MTFaMCMGCSqGSIb3DQEJBDEWBBQeT4GI2yzNyJQ+j3ndSF0MUyqM czBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFlAwQBKjALBglghkgBZQMEAQIwCgYIKoZIhvcN AwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMC AgEoMIGlBgkrBgEEAYI3EAQxgZcwgZQwgYwxCzAJBgNVBAYTAklMMRYwFAYDVQQKEw1TdGFy dENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUgRGlnaXRhbCBDZXJ0aWZpY2F0ZSBTaWduaW5n MTgwNgYDVQQDEy9TdGFydENvbSBDbGFzcyAxIFByaW1hcnkgSW50ZXJtZWRpYXRlIENsaWVu dCBDQQIDBdMXMIGnBgsqhkiG9w0BCRACCzGBl6CBlDCBjDELMAkGA1UEBhMCSUwxFjAUBgNV BAoTDVN0YXJ0Q29tIEx0ZC4xKzApBgNVBAsTIlNlY3VyZSBEaWdpdGFsIENlcnRpZmljYXRl IFNpZ25pbmcxODA2BgNVBAMTL1N0YXJ0Q29tIENsYXNzIDEgUHJpbWFyeSBJbnRlcm1lZGlh dGUgQ2xpZW50IENBAgMF0xcwDQYJKoZIhvcNAQEBBQAEggEAhA0Zlk/iNJ9f0AWzFcfUsCaK rHYhh+JjElHBEZxTPUFLlmqkeAwqLHhATMKEWOn48vvTB36JStzZYqfIprSzuu+nYAl1V8pr Jcb/yd8JvrqwW3w7FA2nRm9o9Hsfjqq2rztZ3XP0OnRAVvW0koPXgZ6mdWEJv7tH43ts/CNg z+Bs1A5L9UgrbyynDPjxNsVbXQc48hQpF95Q9S3/heJkmNY7dusPWUsmfucl/Vl9Y0NziQkx RVgrZS0X5S1igvii/RNDEEzfjBzwMlFeqYF0Xa6SudAW7ZqTrWlkDFXukyegiA+yqiGiDAK4 FIJzNU1AYInHsMD3oAS52Cfpxw8wBgAAAAAAAA== --------------ms090306080802060506010307-- --===============4170748680092524933== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============4170748680092524933==--