Re: 4.2.1: Poor write performance for DomU.

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Mon, Mar 25, 2013 at 01:21:09PM +1100, Steven Haigh wrote:
> So, based on my tests yesterday, I decided to break the RAID6 and
> pull a drive out of it to test directly on the 2Tb drives in
> question.
> 
> The array in question:
> # cat /proc/mdstat
> Personalities : [raid1] [raid6] [raid5] [raid4]
> md2 : active raid6 sdd[4] sdc[0] sde[1] sdf[5]
>       3907026688 blocks super 1.2 level 6, 128k chunk, algorithm 2
> [4/4] [UUUU]
> 
> # mdadm /dev/md2 --fail /dev/sdf
> mdadm: set /dev/sdf faulty in /dev/md2
> # mdadm /dev/md2 --remove /dev/sdf
> mdadm: hot removed /dev/sdf from /dev/md2
> 
> So, all tests are to be done on /dev/sdf.
> Model Family:     Seagate SV35
> Device Model:     ST2000VX000-9YW164
> Serial Number:    Z1E17C3X
> LU WWN Device Id: 5 000c50 04e1bc6f0
> Firmware Version: CV13
> User Capacity:    2,000,398,934,016 bytes [2.00 TB]
> Sector Sizes:     512 bytes logical, 4096 bytes physical
> 
> From the Dom0:
> # dd if=/dev/zero of=/dev/sdf bs=1M count=4096 oflag=direct
> 4096+0 records in
> 4096+0 records out
> 4294967296 bytes (4.3 GB) copied, 30.7691 s, 140 MB/s
> 
> Create a single partition on the drive, and format it with ext4:
> Disk /dev/sdf: 2000.4 GB, 2000398934016 bytes
> 255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
> Units = sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 4096 bytes
> I/O size (minimum/optimal): 4096 bytes / 4096 bytes
> Disk identifier: 0x98d8baaf
> 
>    Device Boot      Start         End      Blocks   Id  System
> /dev/sdf1            2048  3907029167  1953513560   83  Linux
> 
> Command (m for help): w
> 
> # mkfs.ext4 -j /dev/sdf1
> ......
> Writing inode tables: done
> Creating journal (32768 blocks): done
> Writing superblocks and filesystem accounting information: done
> 
> Mount it on the Dom0:
> # mount /dev/sdf1 /mnt/esata/
> # cd /mnt/esata/
> # bonnie++ -d . -u 0:0
> ....
> Version  1.96       ------Sequential Output------ --Sequential
> Input- --Random-
> Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr-
> --Block-- --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec
> %CP /sec %CP
> xenhost.lan.crc. 2G   425  94 133607  24 60544  12   973  95 209114
> 17 296.4   6
> Latency             70971us     190ms     221ms   40369us   17657us
> 164ms
> 
> So from the Dom0: 133Mb/sec write, 209Mb/sec read.
> 
> Now, I'll attach the full disk to a DomU:
> # xm block-attach zeus.vm phy:/dev/sdf xvdc w
> 
> And we'll test from the DomU.
> 
> # dd if=/dev/zero of=/dev/xvdc bs=1M count=4096 oflag=direct
> 4096+0 records in
> 4096+0 records out
> 4294967296 bytes (4.3 GB) copied, 32.318 s, 133 MB/s
> 
> Partition the same as in the Dom0 and create an ext4 filesystem on it:
> 
> I notice something interesting here. In the Dom0, the device is seen as:
> Units = sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 4096 bytes
> I/O size (minimum/optimal): 4096 bytes / 4096 bytes
> 
> In the DomU, it is seen as:
> Units = sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> 
> Not sure if this could be related - but continuing testing:
>     Device Boot      Start         End      Blocks   Id  System
> /dev/xvdc1            2048  3907029167  1953513560   83  Linux
> 
> # mkfs.ext4 -j /dev/xvdc1
> ....
> Allocating group tables: done
> Writing inode tables: done
> Creating journal (32768 blocks): done
> Writing superblocks and filesystem accounting information: done
> 
> # mount /dev/xvdc1 /mnt/esata/
> # cd /mnt/esata/
> # bonnie++ -d . -u 0:0
> ....
> Version  1.96       ------Sequential Output------ --Sequential
> Input- --Random-
> Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr-
> --Block-- --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec
> %CP /sec %CP
> zeus.crc.id.au   2G   396  99 116530  23 50451  15  1035  99 176407
> 23 313.4   9
> Latency             34615us     130ms     128ms   33316us   74401us
> 130ms
> 
> So still... 116Mb/sec write, 176Mb/sec read to the physical device
> from the DomU. More than acceptable.
> 
> It leaves me to wonder.... Could there be something in the Dom0
> seeing the drives as 4096 byte sectors, but the DomU seeing it as
> 512 byte sectors cause an issue?

There is certain overhead in it. I still have this in my mailbox
so I am not sure whether this issue got ever resolved? I know that the 
indirect patches in Xen blkback and xen blkfront are meant to resolve
some of these issues - by being able to carry a bigger payload.

Did you ever try v3.11 kernel in both dom0 and domU? Thanks.
> 
> -- 
> Steven Haigh
> 
> Email: netwiz@crc.id.au
> Web: https://www.crc.id.au
> Phone: (03) 9001 6090 - 0412 935 897
> Fax: (03) 8338 0299