From mboxrd@z Thu Jan 1 00:00:00 1970 From: Konrad Rzeszutek Wilk Subject: Re: 4.2.1: Poor write performance for DomU. Date: Fri, 8 Mar 2013 15:49:16 -0500 Message-ID: <20130308204916.GB22146@phenom.dumpdata.com> References: <51243089.8020307@crc.id.au> <512488AC.1020702@citrix.com> <51248E23.7060408@crc.id.au> <51249C11.3050800@crc.id.au> <5139A747.80703@crc.id.au> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: Content-Disposition: inline In-Reply-To: <5139A747.80703@crc.id.au> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Steven Haigh Cc: "xen-devel@lists.xen.org" , Roger Pau =?iso-8859-1?Q?Monn=E9?= List-Id: xen-devel@lists.xenproject.org On Fri, Mar 08, 2013 at 07:54:31PM +1100, Steven Haigh wrote: > On 20/02/2013 8:49 PM, Steven Haigh wrote: > >On 20/02/2013 7:49 PM, Steven Haigh wrote: > >>On 20/02/2013 7:26 PM, Roger Pau Monn=E9 wrote: > >>>On 20/02/13 03:10, Steven Haigh wrote: > >>>>Hi guys, > >>>> > >>>>Firstly, please CC me in to any replies as I'm not a subscriber these > >>>>days. > >>>> > >>>>I've been trying to debug a problem with Xen 4.2.1 where I am unable = to > >>>>achieve more than ~50Mb/sec sustained sequential write to a disk. The > >>>>DomU is configured as such: > >>> > >>>Since you mention 4.2.1 explicitly, is this a performance regression > >>>from previous versions? (4.2.0 or the 4.1 branch) > >> > >>This is actually a very good question. I've reinstalled my older > >>packages of Xen 4.1.3 back on the system. Rebooting into the new > >>hypervisor, then starting the single DomU again. Ran bonnie++ again on > >>the DomU: > >> > >>Still around 50Mb/sec - so this doesn't seem to be a regression, but > >>something else? > > > >I've actually done a bit of thinking about this... A recent thread on > >linux-raid kernel mailing list about Xen and DomU throughput made me > >revisit my setup. I know I used to be able to saturate GigE both ways > >(send and receive) to the samba share served by this DomU. This would > >mean I'd get at least 90-100Mbyte/sec. What exact config and kernel/xen > >versions this was as this point in time I cannot say. > > > >As such, I had a bit of a play and recreated my RAID6 with 64Kb chunk > >size. This seemed to make rebuild/resync speeds way worse - so I > >reverted to 128Kb chunk size. > > > >The benchmarks I am getting from the Dom0 is about what I'd expect - but > >I wouldn't expect to lose 130Mb/sec write speed to the phy:/ pass > >through of the LV. > > > > From my known config where I could saturate the GigE connection, I have > >changed from kernel 2.6.32 (Jeremy's git repo) to the latest vanilla > >kernels - currently 3.7.9. > > > >My build of Xen 4.2.1 also has all of the recent security advisories > >patched as well. Although it is interesting to note that downgrading to > >Xen 4.1.2 made no difference to write speeds. > > > = > Just wondering if there is any further news or tests that I might be > able to do on this? So usually the problem like this is to unpeel the layers and find out = which of them is at fault. You have a stacked block system - LVM on top of RAID6 on top of block devices. To figure out who is interferring with the speeds I would recommend you fault one of the RAID6 disks (so take it out of the RAID6). Pass it to the guest as a raw disk (/dev/sdX as /dev/xvd) and then run 'fio'. Run 'fio' as well in dom0 on the /dev/sdX and check whether the write performance is different. This is how I how do it: [/dev/xvdXXX] rw=3Dwrite direct=3D1 size=3D4g ioengine=3Dlibaio iodepth=3D32 Then progress up the stack. Try sticking the disk back in RAID6 and doing it on the RAID6. Then on the LVM and so on.