From mboxrd@z Thu Jan 1 00:00:00 1970 From: Steven Haigh Subject: Re: 4.2.1: Poor write performance for DomU. Date: Thu, 05 Sep 2013 18:28:25 +1000 Message-ID: <522840A9.7020101@crc.id.au> References: <20130308204916.GB22146@phenom.dumpdata.com> <513A669E.4060906@crc.id.au> <20130311133017.GJ23018@phenom.dumpdata.com> <513DDE07.2090607@crc.id.au> <20130312130430.GA17901@phenom.dumpdata.com> <514EA337.7030303@crc.id.au> <514EA6B0.8010504@crc.id.au> <514EA741.7050403@crc.id.au> <514EC323.3040106@crc.id.au> <514FB495.2000200@crc.id.au> <20130820164826.GC11652@phenom.dumpdata.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2834156880818055266==" Return-path: In-Reply-To: <20130820164826.GC11652@phenom.dumpdata.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Konrad Rzeszutek Wilk Cc: roger.pau@citrix.com, xen-devel@lists.xen.org List-Id: xen-devel@lists.xenproject.org This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --===============2834156880818055266== Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="CGpbiEhcfJmbpOsPioG79RjroDuTPV1D1" This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --CGpbiEhcfJmbpOsPioG79RjroDuTPV1D1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 21/08/13 02:48, Konrad Rzeszutek Wilk wrote: > On Mon, Mar 25, 2013 at 01:21:09PM +1100, Steven Haigh wrote: >> So, based on my tests yesterday, I decided to break the RAID6 and >> pull a drive out of it to test directly on the 2Tb drives in >> question. >> >> The array in question: >> # cat /proc/mdstat >> Personalities : [raid1] [raid6] [raid5] [raid4] >> md2 : active raid6 sdd[4] sdc[0] sde[1] sdf[5] >> 3907026688 blocks super 1.2 level 6, 128k chunk, algorithm 2 >> [4/4] [UUUU] >> >> # mdadm /dev/md2 --fail /dev/sdf >> mdadm: set /dev/sdf faulty in /dev/md2 >> # mdadm /dev/md2 --remove /dev/sdf >> mdadm: hot removed /dev/sdf from /dev/md2 >> >> So, all tests are to be done on /dev/sdf. >> Model Family: Seagate SV35 >> Device Model: ST2000VX000-9YW164 >> Serial Number: Z1E17C3X >> LU WWN Device Id: 5 000c50 04e1bc6f0 >> Firmware Version: CV13 >> User Capacity: 2,000,398,934,016 bytes [2.00 TB] >> Sector Sizes: 512 bytes logical, 4096 bytes physical >> >> From the Dom0: >> # dd if=3D/dev/zero of=3D/dev/sdf bs=3D1M count=3D4096 oflag=3Ddirect >> 4096+0 records in >> 4096+0 records out >> 4294967296 bytes (4.3 GB) copied, 30.7691 s, 140 MB/s >> >> Create a single partition on the drive, and format it with ext4: >> Disk /dev/sdf: 2000.4 GB, 2000398934016 bytes >> 255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sector= s >> Units =3D sectors of 1 * 512 =3D 512 bytes >> Sector size (logical/physical): 512 bytes / 4096 bytes >> I/O size (minimum/optimal): 4096 bytes / 4096 bytes >> Disk identifier: 0x98d8baaf >> >> Device Boot Start End Blocks Id System >> /dev/sdf1 2048 3907029167 1953513560 83 Linux >> >> Command (m for help): w >> >> # mkfs.ext4 -j /dev/sdf1 >> ...... >> Writing inode tables: done >> Creating journal (32768 blocks): done >> Writing superblocks and filesystem accounting information: done >> >> Mount it on the Dom0: >> # mount /dev/sdf1 /mnt/esata/ >> # cd /mnt/esata/ >> # bonnie++ -d . -u 0:0 >> .... >> Version 1.96 ------Sequential Output------ --Sequential >> Input- --Random- >> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- >> --Block-- --Seeks-- >> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec >> %CP /sec %CP >> xenhost.lan.crc. 2G 425 94 133607 24 60544 12 973 95 209114 >> 17 296.4 6 >> Latency 70971us 190ms 221ms 40369us 17657us >> 164ms >> >> So from the Dom0: 133Mb/sec write, 209Mb/sec read. >> >> Now, I'll attach the full disk to a DomU: >> # xm block-attach zeus.vm phy:/dev/sdf xvdc w >> >> And we'll test from the DomU. >> >> # dd if=3D/dev/zero of=3D/dev/xvdc bs=3D1M count=3D4096 oflag=3Ddirect= >> 4096+0 records in >> 4096+0 records out >> 4294967296 bytes (4.3 GB) copied, 32.318 s, 133 MB/s >> >> Partition the same as in the Dom0 and create an ext4 filesystem on it:= >> >> I notice something interesting here. In the Dom0, the device is seen a= s: >> Units =3D sectors of 1 * 512 =3D 512 bytes >> Sector size (logical/physical): 512 bytes / 4096 bytes >> I/O size (minimum/optimal): 4096 bytes / 4096 bytes >> >> In the DomU, it is seen as: >> Units =3D sectors of 1 * 512 =3D 512 bytes >> Sector size (logical/physical): 512 bytes / 512 bytes >> I/O size (minimum/optimal): 512 bytes / 512 bytes >> >> Not sure if this could be related - but continuing testing: >> Device Boot Start End Blocks Id System >> /dev/xvdc1 2048 3907029167 1953513560 83 Linux >> >> # mkfs.ext4 -j /dev/xvdc1 >> .... >> Allocating group tables: done >> Writing inode tables: done >> Creating journal (32768 blocks): done >> Writing superblocks and filesystem accounting information: done >> >> # mount /dev/xvdc1 /mnt/esata/ >> # cd /mnt/esata/ >> # bonnie++ -d . -u 0:0 >> .... >> Version 1.96 ------Sequential Output------ --Sequential >> Input- --Random- >> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- >> --Block-- --Seeks-- >> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec >> %CP /sec %CP >> zeus.crc.id.au 2G 396 99 116530 23 50451 15 1035 99 176407 >> 23 313.4 9 >> Latency 34615us 130ms 128ms 33316us 74401us >> 130ms >> >> So still... 116Mb/sec write, 176Mb/sec read to the physical device >> from the DomU. More than acceptable. >> >> It leaves me to wonder.... Could there be something in the Dom0 >> seeing the drives as 4096 byte sectors, but the DomU seeing it as >> 512 byte sectors cause an issue? >=20 > There is certain overhead in it. I still have this in my mailbox > so I am not sure whether this issue got ever resolved? I know that the = > indirect patches in Xen blkback and xen blkfront are meant to resolve > some of these issues - by being able to carry a bigger payload. >=20 > Did you ever try v3.11 kernel in both dom0 and domU? Thanks. Ok, so I finally got around to building kernel 3.11 RPMs today for testing. I upgraded both the Dom0 and DomU to the same kernel: DomU: # dmesg | grep blkfront blkfront: xvda: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled; blkfront: xvdb: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled; Looks good. Transfer tests using bonnie++ as per before: # bonnie -d . -u 0:0 Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP zeus.crc.id.au 2G 603 92 58250 9 62248 14 886 99 295757 30 492.3 13 Latency 27305us 124ms 158ms 34222us 16865us 374ms Version 1.96 ------Sequential Create------ --------Random Create-------- zeus.crc.id.au -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 10048 22 +++++ +++ 17849 29 11109 25 +++++ +++ 18389 31 Latency 17775us 154us 180us 16008us 38us 58us Still seems to be a massive discrepancy between Dom0 and DomU write speeds. Interesting is that sequential block reads are nearly 300MB/sec, yet sequential writes were only ~58MB/sec. --=20 Steven Haigh Email: netwiz@crc.id.au Web: https://www.crc.id.au Phone: (03) 9001 6090 - 0412 935 897 Fax: (03) 8338 0299 --CGpbiEhcfJmbpOsPioG79RjroDuTPV1D1 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (GNU/Linux) iQIcBAEBAgAGBQJSKECpAAoJEEGvNdV6fTHc4vUP/10zHpMxg8DibuZAb10T6F77 fuKbnthCxUNn1S0MDGfwEUyXMAshJ3pTwb5IdUSXTjHa5pHcclZXVBIhRlk3Gs0d fKNqerA3KreYJBdnKLXKKtjRZyl1BpKSSCl1OekDmP4/Xnt7vfwQu+P9zlMUnBBs Sg+q2CFlEAotB7R1F40pmHRxey1BpubD9IERfyPddGMGMDdTpl6SZ3DuMkTE9b4E 77v23K47pk5pDgD0x8Pp03Gf+3zWmfQgam2zX8puEsV7/eR659OYi1RWqmz8cFHz vf3Kwa3ymn7cenl2LxQ0tJ0eD8QgwF4F5lOUY448bk/Zh+oAHw9lkQUW/KexqyKh 2dGGa0jXFa3v+IISiBpXUQZTEcuNIPkvky41oiYWRX50r9UB99zEkl5SK+TZZFBZ LHWpRK0cmQn4vN78LJy6E+kZPLO0/qpWpeu09QEGMIG/entBFhVvOpH7S9qAdS5/ 9nbji6C5lvgvIT16UBuX323stA7AAU/CZ6E0Sv9j/4d54YH1+6pWfT1b1n7Fp1+Q i86U4BjsRVGhOJkNrFIJEsCkHDXkVzk4zM+0WxlmaGUNBrkYVUYA5YXcJDxLe1dh cBeSgGU8Bv7PgRhc+3attt37WQaSjiLgkAoE9fEBh+KHIWC3Do4ivrYg1UPt6AbW Sy7pOzrfNf97pLSAXhZ5 =MJQ/ -----END PGP SIGNATURE----- --CGpbiEhcfJmbpOsPioG79RjroDuTPV1D1-- --===============2834156880818055266== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============2834156880818055266==--