From mboxrd@z Thu Jan 1 00:00:00 1970 From: Steven Haigh Subject: 4.2.1: Poor write performance for DomU. Date: Wed, 20 Feb 2013 13:10:17 +1100 Message-ID: <51243089.8020307@crc.id.au> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============6982867781443144015==" Return-path: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: xen-devel@lists.xen.org List-Id: xen-devel@lists.xenproject.org This is a cryptographically signed message in MIME format. --===============6982867781443144015== Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha1; boundary="------------ms080908050302020200070204" This is a cryptographically signed message in MIME format. --------------ms080908050302020200070204 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable Hi guys, Firstly, please CC me in to any replies as I'm not a subscriber these day= s. I've been trying to debug a problem with Xen 4.2.1 where I am unable to=20 achieve more than ~50Mb/sec sustained sequential write to a disk. The=20 DomU is configured as such: name =3D "zeus.vm" memory =3D 1024 vcpus =3D 2 cpus =3D "1-3" disk =3D [ 'phy:/dev/RAID1/zeus.vm,xvda,w',=20 'phy:/dev/vg_raid6/fileshare,xvdb,w' ] vif =3D [ "mac=3D02:16:36:35:35:09, bridge=3Dbr203,=20 vifname=3Dvm.zeus.203", "mac=3D10:16:36:35:35:09, bridge=3Dbr10,=20 vifname=3Dvm.zeus.10" ] bootloader =3D "pygrub" on_poweroff =3D 'destroy' on_reboot =3D 'restart' on_crash =3D 'restart' I have tested the underlying LVM config by mounting=20 /dev/vg_raid6/fileshare from within the Dom0 and running bonnie++ as a=20 benchmark: Version 1.96 ------Sequential Output------ --Sequential Input-=20 --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--=20 --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP=20 /sec %CP xenhost.lan.crc. 2G 667 96 186976 21 80430 14 956 95 290591 26=20 373.7 8 Latency 26416us 212ms 168ms 35494us 35989us 83759= us Version 1.96 ------Sequential Create------ --------Random=20 Create-------- xenhost.lan.crc.id. -Create-- --Read--- -Delete-- -Create-- --Read---=20 -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP=20 /sec %CP 16 14901 32 +++++ +++ 19672 39 15307 34 +++++ +++=20 18158 37 Latency 17838us 141us 298us 365us 133us 296us= ~186Mb/sec write, ~290Mb/sec read. Awesome. I then started a single DomU which gets passed /dev/vg_raid6/fileshare=20 through as xvdb. It is then mounted in /mnt/fileshare/. I then ran=20 bonnie++ again in the DomU: Version 1.96 ------Sequential Output------ --Sequential Input-=20 --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--=20 --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP=20 /sec %CP zeus.crc.id.au 2G 658 96 50618 8 42398 10 1138 99 267568 30=20 494.9 11 Latency 22959us 226ms 311ms 14617us 41816us 72814= us Version 1.96 ------Sequential Create------ --------Random=20 Create-------- zeus.crc.id.au -Create-- --Read--- -Delete-- -Create-- --Read---=20 -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP=20 /sec %CP 16 21749 59 +++++ +++ 31089 73 23283 64 +++++ +++=20 31114 75 Latency 18989us 164us 928us 480us 26us 87us= ~50Mb/sec write, ~267Mb/sec read. Not so awesome. /dev/vg_raid6/fileshare exists as an LV on /dev/md2: # lvdisplay vg_raid6/fileshare --- Logical volume --- LV Path /dev/vg_raid6/fileshare LV Name fileshare VG Name vg_raid6 LV UUID cwC0yK-Xr56-WB5v-10bw-3AZT-pYj0-piWett LV Write Access read/write LV Creation host, time xenhost.lan.crc.id.au, 2013-02-18 20:59:40 +110= 0 LV Status available # open 1 LV Size 2.50 TiB Current LE 655360 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 1024 Block device 253:5 md2 : active raid6 sdd[4] sdc[0] sde[1] sdf[5] 3907026688 blocks super 1.2 level 6, 128k chunk, algorithm 2=20 [4/4] [UUUU] Heres a quick output of 'xm info' - although its full VM load is running = now: # xm info host : xenhost.lan.crc.id.au release : 3.7.9-1.el6xen.x86_64 version : #1 SMP Mon Feb 18 14:46:35 EST 2013 machine : x86_64 nr_cpus : 4 nr_nodes : 1 cores_per_socket : 4 threads_per_core : 1 cpu_mhz : 3303 hw_caps :=20 bfebfbff:28100800:00000000:00003f40:179ae3bf:00000000:00000001:00000000 virt_caps : hvm total_memory : 8116 free_memory : 1346 free_cpus : 0 xen_major : 4 xen_minor : 2 xen_extra : .1 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32=20 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=3D0xffff800000000000 xen_changeset : unavailable xen_commandline : dom0_mem=3D1024M cpufreq=3Dxen dom0_max_vcpus=3D= 1=20 dom0_vcpus_pin cc_compiler : gcc (GCC) 4.4.6 20120305 (Red Hat 4.4.6-4) cc_compile_by : mockbuild cc_compile_domain : crc.id.au cc_compile_date : Sat Feb 16 19:16:38 EST 2013 xend_config_format : 4 In a nutshell, does anyone know *why* I am only able to get ~50Mb/sec=20 sequential writes to the DomU? It certainly isn't a problem getting=20 normal speeds to the LV while mounted in the Dom0. All OS are Scientific Linux 6.3. The Dom0 runs packages from my=20 kernel-xen repo (http://au1.mirror.crc.id.au/repo/el6/x86_64/). The DomU = is completely stock packages. --=20 Steven Haigh Email: netwiz@crc.id.au Web: http://www.crc.id.au Phone: (03) 9001 6090 - 0412 935 897 Fax: (03) 8338 0299 --------------ms080908050302020200070204 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIMdTCC BjQwggQcoAMCAQICAR4wDQYJKoZIhvcNAQEFBQAwfTELMAkGA1UEBhMCSUwxFjAUBgNVBAoT DVN0YXJ0Q29tIEx0ZC4xKzApBgNVBAsTIlNlY3VyZSBEaWdpdGFsIENlcnRpZmljYXRlIFNp Z25pbmcxKTAnBgNVBAMTIFN0YXJ0Q29tIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MB4XDTA3 MTAyNDIxMDE1NVoXDTE3MTAyNDIxMDE1NVowgYwxCzAJBgNVBAYTAklMMRYwFAYDVQQKEw1T dGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUgRGlnaXRhbCBDZXJ0aWZpY2F0ZSBTaWdu aW5nMTgwNgYDVQQDEy9TdGFydENvbSBDbGFzcyAxIFByaW1hcnkgSW50ZXJtZWRpYXRlIENs aWVudCBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMcJg8zOLdgasSmkLhOr lr6KMoOMpohBllVHrdRvEg/q6r8jR+EK75xCGhR8ToREoqe7zM9/UnC6TS2y9UKTpT1v7RSM zR0t6ndl0TWBuUr/UXBhPk+Kmy7bI4yW4urC+y7P3/1/X7U8ocb8VpH/Clt+4iq7nirMcNh6 qJR+xjOhV+VHzQMALuGYn5KZmc1NbJQYclsGkDxDz2UbFqE2+6vIZoL+jb9x4Pa5gNf1TwSD kOkikZB1xtB4ZqtXThaABSONdfmv/Z1pua3FYxnCFmdr/+N2JLKutIxMYqQOJebr/f/h5t95 m4JgrM3Y/w7YX9d7YAL9jvN4SydHsU6n65cCAwEAAaOCAa0wggGpMA8GA1UdEwEB/wQFMAMB Af8wDgYDVR0PAQH/BAQDAgEGMB0GA1UdDgQWBBRTcu2SnODaywFcfH6WNU7y1LhRgjAfBgNV HSMEGDAWgBROC+8apEBbpRdphzDKNGhD0EGu8jBmBggrBgEFBQcBAQRaMFgwJwYIKwYBBQUH MAGGG2h0dHA6Ly9vY3NwLnN0YXJ0c3NsLmNvbS9jYTAtBggrBgEFBQcwAoYhaHR0cDovL3d3 dy5zdGFydHNzbC5jb20vc2ZzY2EuY3J0MFsGA1UdHwRUMFIwJ6AloCOGIWh0dHA6Ly93d3cu c3RhcnRzc2wuY29tL3Nmc2NhLmNybDAnoCWgI4YhaHR0cDovL2NybC5zdGFydHNzbC5jb20v c2ZzY2EuY3JsMIGABgNVHSAEeTB3MHUGCysGAQQBgbU3AQIBMGYwLgYIKwYBBQUHAgEWImh0 dHA6Ly93d3cuc3RhcnRzc2wuY29tL3BvbGljeS5wZGYwNAYIKwYBBQUHAgEWKGh0dHA6Ly93 d3cuc3RhcnRzc2wuY29tL2ludGVybWVkaWF0ZS5wZGYwDQYJKoZIhvcNAQEFBQADggIBAAqD CH14qywGXLhjjF6uHLkjd02hcdh9hrw+VUsv+q1eeQWB21jWj3kJ96AUlPCoEGZ/ynJNScWy 6QMVQjbbMXltUfO4n4bGGdKo3awPWp61tjAFgraLJgDk+DsSvUD6EowjMTNx25GQgyYJ5RPI zKKR9tQW8gGK+2+RHxkUCTbYFnL6kl8Ch507rUdPPipJ9CgJFws3kDS3gOS5WFMxcjO5DwKf KSETEPrHh7p5shuuNktvsv6hxHTLhiMKX893gxdT3XLS9OKmCv87vkINQcNEcIIoFWbP9HOR z9v3vQwR4e3ksLc2JZOAFK+ssS5XMEoznzpihEP0PLc4dCBYjbvSD7kxgDwZ+Aj8Q9PkbvE9 sIPP7ON0fz095HdThKjiVJe6vofq+n6b1NBc8XdrQvBmunwxD5nvtTW4vtN6VY7mUCmxsCie uoBJ9OlqmsVWQvifIYf40dJPZkk9YgGTzWLpXDSfLSplbY2LL9C9U0ptvjcDjefLTvqSFc7t w1sEhF0n/qpA2r0GpvkLRDmcSwVyPvmjFBGqUp/pNy8ZuPGQmHwFi2/14+xeSUDG2bwnsYJQ G2EdJCB6luQ57GEnTA/yKZSTKI8dDQa8Sd3zfXb19mOgSF0bBdXbuKhEpuP9wirslFe6fQ1t 5j5R0xi72MZ8ikMu1RQZKCyDbMwazlHiMIIGOTCCBSGgAwIBAgIDBdMXMA0GCSqGSIb3DQEB BQUAMIGMMQswCQYDVQQGEwJJTDEWMBQGA1UEChMNU3RhcnRDb20gTHRkLjErMCkGA1UECxMi U2VjdXJlIERpZ2l0YWwgQ2VydGlmaWNhdGUgU2lnbmluZzE4MDYGA1UEAxMvU3RhcnRDb20g Q2xhc3MgMSBQcmltYXJ5IEludGVybWVkaWF0ZSBDbGllbnQgQ0EwHhcNMTMwMTI5MTIyMTM0 WhcNMTQwMTMwMjIzNjE4WjBXMRkwFwYDVQQNExA0MzM5TjFVWjZFS2ZBa0Y2MRkwFwYDVQQD DBBuZXR3aXpAY3JjLmlkLmF1MR8wHQYJKoZIhvcNAQkBFhBuZXR3aXpAY3JjLmlkLmF1MIIB IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAqz7eWA7Oi1ES6IIWwtRNtOa6cSj6GSio 72D26TkCPm/vYamqS/9pluIyMO2TbGjMyveGxWZB9bDt+MnuI6yJ3spD6br5TUXQSqsgp1/g NXicgNmUHESd/A7+POEdYKMyTtXjVEqmR8R6uPtD78MBhIKQVZdBYGFuRBhrjoFMOnWARmXD 47FG6eLH8jyeBi+RGEYEpBgTe1gFcoU1dCrjBqxAj47ALNgcHP4ffNX47DLvyhW2pMMaOYUh ga/UyQuPOpZcMAJdVI7dTIkbdipboEEE5pTdoIRI+F+XDLzO8bNkGLa1xLTWrbP048xxw3SL Aw/DmLYqQ+oC8vabAAqBIwIDAQABo4IC1jCCAtIwCQYDVR0TBAIwADALBgNVHQ8EBAMCBLAw HQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMB0GA1UdDgQWBBR/FxImYCufSdz0pG5w A8VPzNyaujAfBgNVHSMEGDAWgBRTcu2SnODaywFcfH6WNU7y1LhRgjAbBgNVHREEFDASgRBu ZXR3aXpAY3JjLmlkLmF1MIIBTAYDVR0gBIIBQzCCAT8wggE7BgsrBgEEAYG1NwECAzCCASow LgYIKwYBBQUHAgEWImh0dHA6Ly93d3cuc3RhcnRzc2wuY29tL3BvbGljeS5wZGYwgfcGCCsG AQUFBwICMIHqMCcWIFN0YXJ0Q29tIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MAMCAQEagb5U aGlzIGNlcnRpZmljYXRlIHdhcyBpc3N1ZWQgYWNjb3JkaW5nIHRvIHRoZSBDbGFzcyAxIFZh bGlkYXRpb24gcmVxdWlyZW1lbnRzIG9mIHRoZSBTdGFydENvbSBDQSBwb2xpY3ksIHJlbGlh bmNlIG9ubHkgZm9yIHRoZSBpbnRlbmRlZCBwdXJwb3NlIGluIGNvbXBsaWFuY2Ugb2YgdGhl IHJlbHlpbmcgcGFydHkgb2JsaWdhdGlvbnMuMDYGA1UdHwQvMC0wK6ApoCeGJWh0dHA6Ly9j cmwuc3RhcnRzc2wuY29tL2NydHUxLWNybC5jcmwwgY4GCCsGAQUFBwEBBIGBMH8wOQYIKwYB BQUHMAGGLWh0dHA6Ly9vY3NwLnN0YXJ0c3NsLmNvbS9zdWIvY2xhc3MxL2NsaWVudC9jYTBC BggrBgEFBQcwAoY2aHR0cDovL2FpYS5zdGFydHNzbC5jb20vY2VydHMvc3ViLmNsYXNzMS5j bGllbnQuY2EuY3J0MCMGA1UdEgQcMBqGGGh0dHA6Ly93d3cuc3RhcnRzc2wuY29tLzANBgkq hkiG9w0BAQUFAAOCAQEAwsdSJioQIH5mEXlMBkQBZN9qrqCvDn4e92NKIX2LMPhVH01o0weH MEgkFMOBypbcPrwnZD51IL+ZfPmuDrcFJR9SPk4W95p3wLakMUBmvDPJ3IC+pa97+fU/GYsh q4dth+VlAIWJarpPV5ee6hAXGQEZvC2jyy5ukcjlr/oQxegb4hufgkLhKjqaq5UDs2be5q1N CBewezVmAbb/VDNVogP1xacnHn4rhllskMfGAda2DPCCaSMaJ3LcC5JRyfMkpKe5UxpVmVeM vGP8hYGI9DlFjwpqRCmIvD/Wl6oWWJc0HaLl6k+U3dngqEHhunzh3H6b/3lWOmEKQuI3Piy+ oDGCA90wggPZAgEBMIGUMIGMMQswCQYDVQQGEwJJTDEWMBQGA1UEChMNU3RhcnRDb20gTHRk LjErMCkGA1UECxMiU2VjdXJlIERpZ2l0YWwgQ2VydGlmaWNhdGUgU2lnbmluZzE4MDYGA1UE AxMvU3RhcnRDb20gQ2xhc3MgMSBQcmltYXJ5IEludGVybWVkaWF0ZSBDbGllbnQgQ0ECAwXT FzAJBgUrDgMCGgUAoIICHTAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJ BTEPFw0xMzAyMjAwMjEwMTdaMCMGCSqGSIb3DQEJBDEWBBS+6iFaXEipT3Hj2AzMVjyp6ZLu jTBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFlAwQBKjALBglghkgBZQMEAQIwCgYIKoZIhvcN AwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMC AgEoMIGlBgkrBgEEAYI3EAQxgZcwgZQwgYwxCzAJBgNVBAYTAklMMRYwFAYDVQQKEw1TdGFy dENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUgRGlnaXRhbCBDZXJ0aWZpY2F0ZSBTaWduaW5n MTgwNgYDVQQDEy9TdGFydENvbSBDbGFzcyAxIFByaW1hcnkgSW50ZXJtZWRpYXRlIENsaWVu dCBDQQIDBdMXMIGnBgsqhkiG9w0BCRACCzGBl6CBlDCBjDELMAkGA1UEBhMCSUwxFjAUBgNV BAoTDVN0YXJ0Q29tIEx0ZC4xKzApBgNVBAsTIlNlY3VyZSBEaWdpdGFsIENlcnRpZmljYXRl IFNpZ25pbmcxODA2BgNVBAMTL1N0YXJ0Q29tIENsYXNzIDEgUHJpbWFyeSBJbnRlcm1lZGlh dGUgQ2xpZW50IENBAgMF0xcwDQYJKoZIhvcNAQEBBQAEggEATNVeiL5uMMyei1IPr1FjZ5v7 l23PCqiA6BsN9eK9gYIwVre9NZA6vDkBga6pz3cMxMJcGu4XFb1PWmq4o9gc+0wWItaCFjID g93/8n0LO6+wMIc1GedGh6wO2DzG6xgGnVIwoBDYJVHQ/biZ+CZ8tqQ9hHqA8qf2Y3f0rR8r 5oRqOaXkBMBBKI59+g2geiA924eX7lhn73WHYJteLxvxcpPkk7SQLlPz+Ay4w76wMQv8lHaG jLnhdUCNmOe7/iwAlg4DE7Er+ZkzKz59oZNM4nkOfurqJEXXg+sDiysLBBwshf3rCGOAGfdf CnQ0WB9M6JCJDgHjYKGnOOu9CGQbdwAAAAAAAA== --------------ms080908050302020200070204-- --===============6982867781443144015== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============6982867781443144015==--