Re: 4.2.1: Poor write performance for DomU.

[Steven Haigh <netwiz@crc.id.au>]

[-- Attachment #1.1: Type: text/plain, Size: 6577 bytes --]

On 21/08/13 02:48, Konrad Rzeszutek Wilk wrote:
> On Mon, Mar 25, 2013 at 01:21:09PM +1100, Steven Haigh wrote:
>> So, based on my tests yesterday, I decided to break the RAID6 and
>> pull a drive out of it to test directly on the 2Tb drives in
>> question.
>>
>> The array in question:
>> # cat /proc/mdstat
>> Personalities : [raid1] [raid6] [raid5] [raid4]
>> md2 : active raid6 sdd[4] sdc[0] sde[1] sdf[5]
>>       3907026688 blocks super 1.2 level 6, 128k chunk, algorithm 2
>> [4/4] [UUUU]
>>
>> # mdadm /dev/md2 --fail /dev/sdf
>> mdadm: set /dev/sdf faulty in /dev/md2
>> # mdadm /dev/md2 --remove /dev/sdf
>> mdadm: hot removed /dev/sdf from /dev/md2
>>
>> So, all tests are to be done on /dev/sdf.
>> Model Family:     Seagate SV35
>> Device Model:     ST2000VX000-9YW164
>> Serial Number:    Z1E17C3X
>> LU WWN Device Id: 5 000c50 04e1bc6f0
>> Firmware Version: CV13
>> User Capacity:    2,000,398,934,016 bytes [2.00 TB]
>> Sector Sizes:     512 bytes logical, 4096 bytes physical
>>
>> From the Dom0:
>> # dd if=/dev/zero of=/dev/sdf bs=1M count=4096 oflag=direct
>> 4096+0 records in
>> 4096+0 records out
>> 4294967296 bytes (4.3 GB) copied, 30.7691 s, 140 MB/s
>>
>> Create a single partition on the drive, and format it with ext4:
>> Disk /dev/sdf: 2000.4 GB, 2000398934016 bytes
>> 255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
>> Units = sectors of 1 * 512 = 512 bytes
>> Sector size (logical/physical): 512 bytes / 4096 bytes
>> I/O size (minimum/optimal): 4096 bytes / 4096 bytes
>> Disk identifier: 0x98d8baaf
>>
>>    Device Boot      Start         End      Blocks   Id  System
>> /dev/sdf1            2048  3907029167  1953513560   83  Linux
>>
>> Command (m for help): w
>>
>> # mkfs.ext4 -j /dev/sdf1
>> ......
>> Writing inode tables: done
>> Creating journal (32768 blocks): done
>> Writing superblocks and filesystem accounting information: done
>>
>> Mount it on the Dom0:
>> # mount /dev/sdf1 /mnt/esata/
>> # cd /mnt/esata/
>> # bonnie++ -d . -u 0:0
>> ....
>> Version  1.96       ------Sequential Output------ --Sequential
>> Input- --Random-
>> Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr-
>> --Block-- --Seeks--
>> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec
>> %CP /sec %CP
>> xenhost.lan.crc. 2G   425  94 133607  24 60544  12   973  95 209114
>> 17 296.4   6
>> Latency             70971us     190ms     221ms   40369us   17657us
>> 164ms
>>
>> So from the Dom0: 133Mb/sec write, 209Mb/sec read.
>>
>> Now, I'll attach the full disk to a DomU:
>> # xm block-attach zeus.vm phy:/dev/sdf xvdc w
>>
>> And we'll test from the DomU.
>>
>> # dd if=/dev/zero of=/dev/xvdc bs=1M count=4096 oflag=direct
>> 4096+0 records in
>> 4096+0 records out
>> 4294967296 bytes (4.3 GB) copied, 32.318 s, 133 MB/s
>>
>> Partition the same as in the Dom0 and create an ext4 filesystem on it:
>>
>> I notice something interesting here. In the Dom0, the device is seen as:
>> Units = sectors of 1 * 512 = 512 bytes
>> Sector size (logical/physical): 512 bytes / 4096 bytes
>> I/O size (minimum/optimal): 4096 bytes / 4096 bytes
>>
>> In the DomU, it is seen as:
>> Units = sectors of 1 * 512 = 512 bytes
>> Sector size (logical/physical): 512 bytes / 512 bytes
>> I/O size (minimum/optimal): 512 bytes / 512 bytes
>>
>> Not sure if this could be related - but continuing testing:
>>     Device Boot      Start         End      Blocks   Id  System
>> /dev/xvdc1            2048  3907029167  1953513560   83  Linux
>>
>> # mkfs.ext4 -j /dev/xvdc1
>> ....
>> Allocating group tables: done
>> Writing inode tables: done
>> Creating journal (32768 blocks): done
>> Writing superblocks and filesystem accounting information: done
>>
>> # mount /dev/xvdc1 /mnt/esata/
>> # cd /mnt/esata/
>> # bonnie++ -d . -u 0:0
>> ....
>> Version  1.96       ------Sequential Output------ --Sequential
>> Input- --Random-
>> Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr-
>> --Block-- --Seeks--
>> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec
>> %CP /sec %CP
>> zeus.crc.id.au   2G   396  99 116530  23 50451  15  1035  99 176407
>> 23 313.4   9
>> Latency             34615us     130ms     128ms   33316us   74401us
>> 130ms
>>
>> So still... 116Mb/sec write, 176Mb/sec read to the physical device
>> from the DomU. More than acceptable.
>>
>> It leaves me to wonder.... Could there be something in the Dom0
>> seeing the drives as 4096 byte sectors, but the DomU seeing it as
>> 512 byte sectors cause an issue?
> 
> There is certain overhead in it. I still have this in my mailbox
> so I am not sure whether this issue got ever resolved? I know that the 
> indirect patches in Xen blkback and xen blkfront are meant to resolve
> some of these issues - by being able to carry a bigger payload.
> 
> Did you ever try v3.11 kernel in both dom0 and domU? Thanks.

Ok, so I finally got around to building kernel 3.11 RPMs today for
testing. I upgraded both the Dom0 and DomU to the same kernel:

DomU:
# dmesg | grep blkfront
blkfront: xvda: flush diskcache: enabled; persistent grants: enabled;
indirect descriptors: enabled;
blkfront: xvdb: flush diskcache: enabled; persistent grants: enabled;
indirect descriptors: enabled;

Looks good.

Transfer tests using bonnie++ as per before:
# bonnie -d . -u 0:0
Version  1.96       ------Sequential Output------ --Sequential Input-
--Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
zeus.crc.id.au   2G   603  92 58250   9 62248  14   886  99 295757  30
492.3  13
Latency             27305us     124ms     158ms   34222us   16865us
374ms
Version  1.96       ------Sequential Create------ --------Random
Create--------
zeus.crc.id.au      -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
/sec %CP
                 16 10048  22 +++++ +++ 17849  29 11109  25 +++++ +++
18389  31
Latency             17775us     154us     180us   16008us      38us
 58us

Still seems to be a massive discrepancy between Dom0 and DomU write
speeds. Interesting is that sequential block reads are nearly 300MB/sec,
yet sequential writes were only ~58MB/sec.

-- 
Steven Haigh

Email: netwiz@crc.id.au
Web: https://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897
Fax: (03) 8338 0299


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel