From mboxrd@z Thu Jan 1 00:00:00 1970 From: Steven Haigh Subject: Re: 4.2.1: Poor write performance for DomU. Date: Wed, 13 Mar 2013 01:08:21 +1100 Message-ID: <513F36D5.6050707@crc.id.au> References: <51243089.8020307@crc.id.au> <512488AC.1020702@citrix.com> <51248E23.7060408@crc.id.au> <51249C11.3050800@crc.id.au> <5139A747.80703@crc.id.au> <20130308204916.GB22146@phenom.dumpdata.com> <513A669E.4060906@crc.id.au> <20130311133017.GJ23018@phenom.dumpdata.com> <513DDE07.2090607@crc.id.au> <20130312130430.GA17901@phenom.dumpdata.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20130312130430.GA17901@phenom.dumpdata.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: xen-devel@lists.xen.org List-Id: xen-devel@lists.xenproject.org On 13/03/13 00:04, Konrad Rzeszutek Wilk wrote: >>> So still filesystem. Fio can do it on a block level. >>> >>> What does 'xenstore-ls' show you and 'losetup -a'? I am really >>> curious as to where that file you are providing to the guest as >>> disk is being handled via 'loop' or via 'QEMU'. >>> >> >> I've picked out what I believe is the most relevant from xenstore-ls >> that belongs to the DomU in question: > > Great. > .. snip.. >> params = "/dev/vg_raid6/fileshare" >> mode = "w" >> online = "1" >> frontend-id = "1" >> type = "phy" >> physical-device = "fd:5" >> hotplug-status = "connected" >> feature-flush-cache = "1" >> feature-discard = "0" >> feature-barrier = "1" >> feature-persistent = "1" >> sectors = "5368709120" >> info = "0" >> sector-size = "512" > > OK, so the flow of data from the guest is: > bonnie++ -> FS -> xen-blkfront -> xen-blkback -> LVM -> RAID6 -> multiple disks. > > Any way you can restructure this to be: > > fio -> xen-blkfront -> xen-blkback -> one disk from the raid. > > > to see if the issue is between "LVM -> RAID6" or the "bonnie++ -> FS" part? > Is the cpu load quite high when you do these writes? Maybe I'm missing something, but running this directly from the Dom0 would give a result of: bonnie++ -> FS -> LVM -> RAID6 These figures were well over 200Mb/sec read and well over 100Mb/sec write. This only takes out the xen-blkfront and xen-blkback - which I thought was the aim? Or is the point of this to make sure that we can replicate it with a single disk and that it isn't some weird interaction between blkfront/blkback and the LVM/RAID6? CPU Usage doesn't seem to be a limiting factor. I certainly don't see massive loads for writing. > > What are the RAID6 disks you have? How many? The RAID6 is made up of 4 x 2Tb 7200RPM Seagate SATA drives... Model Family: Seagate SV35 Device Model: ST2000VX000-9YW164 Serial Number: Z1E10QQJ LU WWN Device Id: 5 000c50 04dd3a1f1 Firmware Version: CV13 User Capacity: 2,000,398,934,016 bytes [2.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Then in /proc/mdstat: md2 : active raid6 sdd[4] sdc[0] sdf[5] sde[1] 3907026688 blocks super 1.2 level 6, 128k chunk, algorithm 2 [4/4] [UUUU] I decided to use whole disks so that I don't run into alignment issues. The VG is using 4Mb extents, so that should be fine too: # vgdisplay vg_raid6 --- Volume group --- VG Name vg_raid6 System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 7 VG Access read/write VG Status resizable MAX LV 0 Cur LV 5 Open LV 5 Max PV 0 Cur PV 1 Act PV 1 VG Size 3.64 TiB PE Size 4.00 MiB Total PE 953863 Alloc PE / Size 688640 / 2.63 TiB Free PE / Size 265223 / 1.01 TiB VG UUID md7G8X-F2mT-JBQa-f5qm-TN4O-kOqs-KWHGR1 -- Steven Haigh Email: netwiz@crc.id.au Web: https://www.crc.id.au Phone: (03) 9001 6090 - 0412 935 897 Fax: (03) 8338 0299