Re: 4.2.1: Poor write performance for DomU.

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Fri, Mar 08, 2013 at 07:54:31PM +1100, Steven Haigh wrote:
> On 20/02/2013 8:49 PM, Steven Haigh wrote:
> >On 20/02/2013 7:49 PM, Steven Haigh wrote:
> >>On 20/02/2013 7:26 PM, Roger Pau Monné wrote:
> >>>On 20/02/13 03:10, Steven Haigh wrote:
> >>>>Hi guys,
> >>>>
> >>>>Firstly, please CC me in to any replies as I'm not a subscriber these
> >>>>days.
> >>>>
> >>>>I've been trying to debug a problem with Xen 4.2.1 where I am unable to
> >>>>achieve more than ~50Mb/sec sustained sequential write to a disk. The
> >>>>DomU is configured as such:
> >>>
> >>>Since you mention 4.2.1 explicitly, is this a performance regression
> >>>from previous versions? (4.2.0 or the 4.1 branch)
> >>
> >>This is actually a very good question. I've reinstalled my older
> >>packages of Xen 4.1.3 back on the system. Rebooting into the new
> >>hypervisor, then starting the single DomU again. Ran bonnie++ again on
> >>the DomU:
> >>
> >>Still around 50Mb/sec - so this doesn't seem to be a regression, but
> >>something else?
> >
> >I've actually done a bit of thinking about this... A recent thread on
> >linux-raid kernel mailing list about Xen and DomU throughput made me
> >revisit my setup. I know I used to be able to saturate GigE both ways
> >(send and receive) to the samba share served by this DomU. This would
> >mean I'd get at least 90-100Mbyte/sec. What exact config and kernel/xen
> >versions this was as this point in time I cannot say.
> >
> >As such, I had a bit of a play and recreated my RAID6 with 64Kb chunk
> >size. This seemed to make rebuild/resync speeds way worse - so I
> >reverted to 128Kb chunk size.
> >
> >The benchmarks I am getting from the Dom0 is about what I'd expect - but
> >I wouldn't expect to lose 130Mb/sec write speed to the phy:/ pass
> >through of the LV.
> >
> > From my known config where I could saturate the GigE connection, I have
> >changed from kernel 2.6.32 (Jeremy's git repo) to the latest vanilla
> >kernels - currently 3.7.9.
> >
> >My build of Xen 4.2.1 also has all of the recent security advisories
> >patched as well. Although it is interesting to note that downgrading to
> >Xen 4.1.2 made no difference to write speeds.
> >
> 
> Just wondering if there is any further news or tests that I might be
> able to do on this?

So usually the problem like this is to unpeel the layers and find out 
which of them is at fault. You have a stacked block system - LVM on
top of RAID6 on top of block devices.

To figure out who is interferring with the speeds I would recommend
you fault one of the RAID6 disks (so take it out of the RAID6). Pass
it to the guest as a raw disk (/dev/sdX as /dev/xvd) and then
run 'fio'. Run 'fio' as well in dom0 on the /dev/sdX and check
whether the write performance is different.

This is how I how do it:

[/dev/xvdXXX]
rw=write
direct=1
size=4g
ioengine=libaio
iodepth=32

Then progress up the stack. Try sticking the disk back in RAID6
and doing it on the RAID6. Then on the LVM and so on.