From mboxrd@z Thu Jan 1 00:00:00 1970 From: Steven Haigh Subject: Re: 4.2.1: Poor write performance for DomU. Date: Wed, 21 Aug 2013 04:25:14 +1000 Message-ID: <5213B48A.2040605@crc.id.au> References: <20130308204916.GB22146@phenom.dumpdata.com> <513A669E.4060906@crc.id.au> <20130311133017.GJ23018@phenom.dumpdata.com> <513DDE07.2090607@crc.id.au> <20130312130430.GA17901@phenom.dumpdata.com> <514EA337.7030303@crc.id.au> <514EA6B0.8010504@crc.id.au> <514EA741.7050403@crc.id.au> <514EC323.3040106@crc.id.au> <514FB495.2000200@crc.id.au> <20130820164826.GC11652@phenom.dumpdata.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20130820164826.GC11652@phenom.dumpdata.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Konrad Rzeszutek Wilk Cc: roger.pau@citrix.com, xen-devel@lists.xen.org List-Id: xen-devel@lists.xenproject.org On 21/08/13 02:48, Konrad Rzeszutek Wilk wrote: > On Mon, Mar 25, 2013 at 01:21:09PM +1100, Steven Haigh wrote: >> So, based on my tests yesterday, I decided to break the RAID6 and >> pull a drive out of it to test directly on the 2Tb drives in >> question. >> >> The array in question: >> # cat /proc/mdstat >> Personalities : [raid1] [raid6] [raid5] [raid4] >> md2 : active raid6 sdd[4] sdc[0] sde[1] sdf[5] >> 3907026688 blocks super 1.2 level 6, 128k chunk, algorithm 2 >> [4/4] [UUUU] >> >> # mdadm /dev/md2 --fail /dev/sdf >> mdadm: set /dev/sdf faulty in /dev/md2 >> # mdadm /dev/md2 --remove /dev/sdf >> mdadm: hot removed /dev/sdf from /dev/md2 >> >> So, all tests are to be done on /dev/sdf. >> Model Family: Seagate SV35 >> Device Model: ST2000VX000-9YW164 >> Serial Number: Z1E17C3X >> LU WWN Device Id: 5 000c50 04e1bc6f0 >> Firmware Version: CV13 >> User Capacity: 2,000,398,934,016 bytes [2.00 TB] >> Sector Sizes: 512 bytes logical, 4096 bytes physical >> >> From the Dom0: >> # dd if=/dev/zero of=/dev/sdf bs=1M count=4096 oflag=direct >> 4096+0 records in >> 4096+0 records out >> 4294967296 bytes (4.3 GB) copied, 30.7691 s, 140 MB/s >> >> Create a single partition on the drive, and format it with ext4: >> Disk /dev/sdf: 2000.4 GB, 2000398934016 bytes >> 255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors >> Units = sectors of 1 * 512 = 512 bytes >> Sector size (logical/physical): 512 bytes / 4096 bytes >> I/O size (minimum/optimal): 4096 bytes / 4096 bytes >> Disk identifier: 0x98d8baaf >> >> Device Boot Start End Blocks Id System >> /dev/sdf1 2048 3907029167 1953513560 83 Linux >> >> Command (m for help): w >> >> # mkfs.ext4 -j /dev/sdf1 >> ...... >> Writing inode tables: done >> Creating journal (32768 blocks): done >> Writing superblocks and filesystem accounting information: done >> >> Mount it on the Dom0: >> # mount /dev/sdf1 /mnt/esata/ >> # cd /mnt/esata/ >> # bonnie++ -d . -u 0:0 >> .... >> Version 1.96 ------Sequential Output------ --Sequential >> Input- --Random- >> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- >> --Block-- --Seeks-- >> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec >> %CP /sec %CP >> xenhost.lan.crc. 2G 425 94 133607 24 60544 12 973 95 209114 >> 17 296.4 6 >> Latency 70971us 190ms 221ms 40369us 17657us >> 164ms >> >> So from the Dom0: 133Mb/sec write, 209Mb/sec read. >> >> Now, I'll attach the full disk to a DomU: >> # xm block-attach zeus.vm phy:/dev/sdf xvdc w >> >> And we'll test from the DomU. >> >> # dd if=/dev/zero of=/dev/xvdc bs=1M count=4096 oflag=direct >> 4096+0 records in >> 4096+0 records out >> 4294967296 bytes (4.3 GB) copied, 32.318 s, 133 MB/s >> >> Partition the same as in the Dom0 and create an ext4 filesystem on it: >> >> I notice something interesting here. In the Dom0, the device is seen as: >> Units = sectors of 1 * 512 = 512 bytes >> Sector size (logical/physical): 512 bytes / 4096 bytes >> I/O size (minimum/optimal): 4096 bytes / 4096 bytes >> >> In the DomU, it is seen as: >> Units = sectors of 1 * 512 = 512 bytes >> Sector size (logical/physical): 512 bytes / 512 bytes >> I/O size (minimum/optimal): 512 bytes / 512 bytes >> >> Not sure if this could be related - but continuing testing: >> Device Boot Start End Blocks Id System >> /dev/xvdc1 2048 3907029167 1953513560 83 Linux >> >> # mkfs.ext4 -j /dev/xvdc1 >> .... >> Allocating group tables: done >> Writing inode tables: done >> Creating journal (32768 blocks): done >> Writing superblocks and filesystem accounting information: done >> >> # mount /dev/xvdc1 /mnt/esata/ >> # cd /mnt/esata/ >> # bonnie++ -d . -u 0:0 >> .... >> Version 1.96 ------Sequential Output------ --Sequential >> Input- --Random- >> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- >> --Block-- --Seeks-- >> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec >> %CP /sec %CP >> zeus.crc.id.au 2G 396 99 116530 23 50451 15 1035 99 176407 >> 23 313.4 9 >> Latency 34615us 130ms 128ms 33316us 74401us >> 130ms >> >> So still... 116Mb/sec write, 176Mb/sec read to the physical device >> from the DomU. More than acceptable. >> >> It leaves me to wonder.... Could there be something in the Dom0 >> seeing the drives as 4096 byte sectors, but the DomU seeing it as >> 512 byte sectors cause an issue? > > There is certain overhead in it. I still have this in my mailbox > so I am not sure whether this issue got ever resolved? I know that the > indirect patches in Xen blkback and xen blkfront are meant to resolve > some of these issues - by being able to carry a bigger payload. > > Did you ever try v3.11 kernel in both dom0 and domU? Thanks. Hi Konrad, I don't believe I ever fixed it - however I haven't tried kernel 3.11 in Dom0 OR DomU... I'll keep this in my inbox and try to build a 3.11 kernel for both in the near future for testing... -- Steven Haigh Email: netwiz@crc.id.au Web: https://www.crc.id.au Phone: (03) 9001 6090 - 0412 935 897 Fax: (03) 8338 0299