Re: 4.2.1: Poor write performance for DomU.

[Steven Haigh <netwiz@crc.id.au>]

[-- Attachment #1.1: Type: text/plain, Size: 5616 bytes --]

On 9/03/2013 7:49 AM, Konrad Rzeszutek Wilk wrote:
> On Fri, Mar 08, 2013 at 07:54:31PM +1100, Steven Haigh wrote:
>> On 20/02/2013 8:49 PM, Steven Haigh wrote:
>>> On 20/02/2013 7:49 PM, Steven Haigh wrote:
>>>> On 20/02/2013 7:26 PM, Roger Pau Monné wrote:
>>>>> On 20/02/13 03:10, Steven Haigh wrote:
>>>>>> Hi guys,
>>>>>>
>>>>>> Firstly, please CC me in to any replies as I'm not a subscriber these
>>>>>> days.
>>>>>>
>>>>>> I've been trying to debug a problem with Xen 4.2.1 where I am unable to
>>>>>> achieve more than ~50Mb/sec sustained sequential write to a disk. The
>>>>>> DomU is configured as such:
>>>>>
>>>>> Since you mention 4.2.1 explicitly, is this a performance regression
>>>> >from previous versions? (4.2.0 or the 4.1 branch)
>>>>
>>>> This is actually a very good question. I've reinstalled my older
>>>> packages of Xen 4.1.3 back on the system. Rebooting into the new
>>>> hypervisor, then starting the single DomU again. Ran bonnie++ again on
>>>> the DomU:
>>>>
>>>> Still around 50Mb/sec - so this doesn't seem to be a regression, but
>>>> something else?
>>>
>>> I've actually done a bit of thinking about this... A recent thread on
>>> linux-raid kernel mailing list about Xen and DomU throughput made me
>>> revisit my setup. I know I used to be able to saturate GigE both ways
>>> (send and receive) to the samba share served by this DomU. This would
>>> mean I'd get at least 90-100Mbyte/sec. What exact config and kernel/xen
>>> versions this was as this point in time I cannot say.
>>>
>>> As such, I had a bit of a play and recreated my RAID6 with 64Kb chunk
>>> size. This seemed to make rebuild/resync speeds way worse - so I
>>> reverted to 128Kb chunk size.
>>>
>>> The benchmarks I am getting from the Dom0 is about what I'd expect - but
>>> I wouldn't expect to lose 130Mb/sec write speed to the phy:/ pass
>>> through of the LV.
>>>
>>>  From my known config where I could saturate the GigE connection, I have
>>> changed from kernel 2.6.32 (Jeremy's git repo) to the latest vanilla
>>> kernels - currently 3.7.9.
>>>
>>> My build of Xen 4.2.1 also has all of the recent security advisories
>>> patched as well. Although it is interesting to note that downgrading to
>>> Xen 4.1.2 made no difference to write speeds.
>>>
>>
>> Just wondering if there is any further news or tests that I might be
>> able to do on this?
>
> So usually the problem like this is to unpeel the layers and find out
> which of them is at fault. You have a stacked block system - LVM on
> top of RAID6 on top of block devices.
>
> To figure out who is interferring with the speeds I would recommend
> you fault one of the RAID6 disks (so take it out of the RAID6). Pass
> it to the guest as a raw disk (/dev/sdX as /dev/xvd) and then
> run 'fio'. Run 'fio' as well in dom0 on the /dev/sdX and check
> whether the write performance is different.
>
> This is how I how do it:
>
> [/dev/xvdXXX]
> rw=write
> direct=1
> size=4g
> ioengine=libaio
> iodepth=32
>
> Then progress up the stack. Try sticking the disk back in RAID6
> and doing it on the RAID6. Then on the LVM and so on.

I did try to peel it back a single layer at a time. My test was simply 
using the same XFS filesystem in the Dom0 instead of the DomU.

I tested the underlying LVM config by mounting /dev/vg_raid6/fileshare 
from within the Dom0 and running bonnie++ as a benchmark:

Version  1.96       ------Sequential Output------ --Sequential Input- 
--Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP 
/sec %CP
xenhost.lan.crc. 2G   667  96 186976  21 80430  14   956  95 290591  26 
373.7   8
Latency             26416us     212ms     168ms   35494us   35989us 83759us
Version  1.96       ------Sequential Create------ --------Random 
Create--------
xenhost.lan.crc.id. -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP 
/sec %CP
                  16 14901  32 +++++ +++ 19672  39 15307  34 +++++ +++ 
18158  37
Latency             17838us     141us     298us     365us     133us 296us

~186Mb/sec write, ~290Mb/sec read. Awesome.

I then started a single DomU which gets passed /dev/vg_raid6/fileshare 
through as xvdb. It is then mounted in /mnt/fileshare/. I then ran 
bonnie++ again in the DomU:

Version  1.96       ------Sequential Output------ --Sequential Input- 
--Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP 
/sec %CP
zeus.crc.id.au   2G   658  96 50618   8 42398  10  1138  99 267568  30 
494.9  11
Latency             22959us     226ms     311ms   14617us   41816us 72814us
Version  1.96       ------Sequential Create------ --------Random 
Create--------
zeus.crc.id.au      -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP 
/sec %CP
                  16 21749  59 +++++ +++ 31089  73 23283  64 +++++ +++ 
31114  75
Latency             18989us     164us     928us     480us      26us  87us

~50Mb/sec write, ~267Mb/sec read. Not so awesome.

As such, the filesystem, RAID6, etc are completely unchanged. The only 
change is the access method Dom0 vs DomU.

-- 
Steven Haigh

Email: netwiz@crc.id.au
Web: https://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897
Fax: (03) 8338 0299


[-- Attachment #1.2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4240 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel