From: "Roger Pau Monné" <roger.pau@citrix.com>
To: Steven Haigh <netwiz@crc.id.au>
Cc: Felipe Franciosi <felipe.franciosi@citrix.com>,
"xen-devel@lists.xen.org" <xen-devel@lists.xen.org>
Subject: Re: IO speed limited by size of IO request (for RBD driver)
Date: Mon, 29 Apr 2013 10:38:45 +0200 [thread overview]
Message-ID: <517E3195.8090204@citrix.com> (raw)
In-Reply-To: <517B8DE3.90306@crc.id.au>
On 27/04/13 10:35, Steven Haigh wrote:
> On 27/04/2013 5:51 PM, Steven Haigh wrote:
>> On 27/04/2013 5:06 PM, Roger Pau Monné wrote:
>>> On 27/04/13 03:57, Steven Haigh wrote:
>>>> On 27/04/2013 12:16 AM, Steven Haigh wrote:
>>>>> On 27/04/2013 12:06 AM, Roger Pau Monné wrote:
>>>>>> On 23/04/13 21:05, Steven Haigh wrote:
>>>>>>> Sorry - resending this to Felipe as well - as I started talking to
>>>>>>> him
>>>>>>> directly previously.
>>>>>>>
>>>>>>> Felipe, to bring you up to date, I've copied over the blkback
>>>>>>> files from
>>>>>>> Rogers indirect kernel over the vanilla 3.8.8 kernel files, built and
>>>>>>> tested. Results below:
>>>>>>>
>>>>>
>>>>> Bringing this into context in a nutshell - results showed about 5MB/sec
>>>>> improvement when using buffered disk access - totalling ~57MB/sec write
>>>>> speed vs ~98MB/sec when using the oflag=direct flag to dd.
>>>>>
>>>>> When talking about back porting a few indirect patches to mainline
>>>>> blkback (3.8.8 atm):
>>>>>>> On 24/04/2013 4:13 AM, Roger Pau Monné wrote:
>>>>>>>> I think it requires a non-trivial amount of work, what you could do
>>>>>>>> as a
>>>>>>>> test is directly replace the affected files with the ones in my
>>>>>>>> tree, it
>>>>>>>> is not optimal, but I don't think it's going to cause problems,
>>>>>>>> and you
>>>>>>>> could at least see if indirect descriptors solve your problem.
>>>>>>>
>>>>>>> Ok, I copied across those files, built, packaged and installed
>>>>>>> them on
>>>>>>> my Dom0. Good news is that its a little quicker, bad news is not by
>>>>>>> much.
>>>>>>
>>>>>> Could you try increasing xen_blkif_max_segments variable in
>>>>>> xen-blkfront.c to 64 or 128? It is set to 32 by default. You will only
>>>>>> need to recompile the DomU kernel after this, the Dom0 is able to
>>>>>> support up to 256 indirect segments.
>>>>>
>>>>> I'll have to look at this. All DomU's are Scientific Linux 6.4
>>>>> systems -
>>>>> so essentially RHEL6.4 and so on. I haven't built a RH kernel as yet -
>>>>> so I'll have to look at what is involved. It might be as simple as
>>>>> rebuilding a normal SRPM.
>>>>
>>>> Ok, I've had a look at the RH xen-blkfront.c - and I can't see any
>>>> definition of xen_blkif_max_segments - or anything close. I've attached
>>>> the version used in the EL6 kernel from the kernel-2.6.32-358.6.1.el6
>>>> srpm.
>>>>
>>>> Any ideas on where to go from here?
>>>
>>> I thought you were using the 3.8.x kernel inside the DomU also, if you
>>> are not using it, then it's normal that there's no speed difference, you
>>> have a Dom0 kernel that supports indirect descriptors, but your DomU
>>> doesn't. You must use a kernel that supports indirect descriptors in
>>> both Dom0 and DomU in order to make use of this feature.
>>
>> Ahhh - sorry - I should have been clearer - The Dom0 is kernel 3.8.x
>> (3.8.8 right now) - however the DomUs are all stock EL6 kernels.
>>
>> Hmmmm - I believe the kernel I build for Dom0 *should* work as a DomU.
>> I'll do some more experimentation and see if I can get it working
>> properly as a DomU kernel.
>
> Ok - now for testing the basics.
>
> Same tests using vanilla 3.8.8 kernel:
> # dd if=/dev/zero of=output.zero bs=1M count=2048
> 2048+0 records in
> 2048+0 records out
> 2147483648 bytes (2.1 GB) copied, 37.1206 s, 57.9 MB/s
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.00 0.00 2.65 0.00 0.22 97.13
>
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
> avgrq-sz avgqu-sz await svctm %util
> sdf 395.81 2201.32 27.59 156.95 1.65 9.21
> 120.60 1.13 6.12 1.19 22.05
> sde 404.86 2208.83 28.04 157.40 1.69 9.24
> 120.77 1.32 7.15 1.31 24.24
> sdc 435.54 2174.83 30.68 155.63 1.82 9.10
> 120.09 0.97 5.20 1.11 20.64
> sdd 388.96 2177.26 26.71 155.41 1.62 9.11
> 120.74 1.10 6.01 1.30 23.60
> md2 0.00 0.00 0.00 537.31 0.00 17.59
> 67.05 0.00 0.00 0.00 0.00
>
> # dd if=/dev/zero of=output.zero bs=1M count=2048 oflag=direct
> 2048+0 records in
> 2048+0 records out
> 2147483648 bytes (2.1 GB) copied, 25.3928 s, 84.6 MB/s
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.22 0.00 15.74 0.00 0.22 83.81
>
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
> avgrq-sz avgqu-sz await svctm %util
> sdf 336.81 12659.65 10.86 232.59 1.36 50.36
> 435.06 1.00 4.09 2.51 61.06
> sde 1684.04 11000.22 54.32 189.14 6.79 43.71
> 424.80 1.45 5.95 3.50 85.28
> sdc 144.35 11177.61 4.66 238.80 0.58 44.60
> 380.04 0.41 1.70 1.07 26.08
> sdd 20.62 12876.50 0.67 242.79 0.08 51.25
> 431.80 0.45 1.84 1.15 27.92
> md2 0.00 0.00 0.00 2680.71 0.00 86.47
> 66.06 0.00 0.00 0.00 0.00
>
> Installed and rebooted into the patched version I build by just copying
> the blkback files across to the 3.8.8 tree and building:
Did you also copy xen-blkfront?
>
> # dd if=/dev/zero of=output.zero bs=1M count=2048
> 2048+0 records in
> 2048+0 records out
> 2147483648 bytes (2.1 GB) copied, 45.2376 s, 47.5 MB/s
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.00 0.00 1.35 0.00 0.45 98.19
>
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
> avgrq-sz avgqu-sz await svctm %util
> sdd 1340.80 5806.80 158.20 674.40 5.83 25.27
> 76.51 6.00 7.16 0.80 66.90
> sde 1334.60 5894.00 160.80 686.40 5.86 25.71
> 76.32 6.87 8.11 0.87 73.52
> sdc 1330.80 5858.20 158.00 682.40 5.86 25.60
> 76.67 5.71 6.81 0.77 64.84
> sdf 1341.00 5848.80 157.00 681.20 5.83 25.49
> 76.53 6.23 7.38 0.85 70.92
> md2 0.00 0.00 0.00 1431.40 0.00 46.83
> 67.01 0.00 0.00 0.00 0.00
>
> # dd if=/dev/zero of=output.zero bs=1M count=2048 oflag=direct
> 2048+0 records in
> 2048+0 records out
> 2147483648 bytes (2.1 GB) copied, 38.9052 s, 55.2 MB/s
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.00 0.00 5.27 0.00 0.32 94.41
>
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
> avgrq-sz avgqu-sz await svctm %util
> sdd 493.20 8481.60 36.80 335.80 2.07 34.45
> 200.73 1.14 3.07 0.97 36.32
> sde 1371.60 7380.20 83.80 304.80 5.66 30.08
> 188.34 2.20 5.65 1.94 75.38
> sdc 540.20 7556.80 56.00 326.20 2.33 30.80
> 177.52 1.49 3.90 1.26 48.02
> sdf 734.20 8286.60 64.40 326.20 3.12 33.67
> 192.92 1.66 4.24 1.45 56.66
> md2 0.00 0.00 0.00 1835.20 0.00 59.20
> 66.06 0.00 0.00 0.00 0.00
>
> That is with the same kernel running on both Dom0 and DomU.
>
> In the dmesg of the DomU, I see the following:
> blkfront: xvdb: flush diskcache: enabled using persistent grants
It seems you are missing some pieces, you should see something like:
blkfront: xvda: flush diskcache: enabled; persistent grants: enabled;
indirect descriptors: enabled;
next prev parent reply other threads:[~2013-04-29 8:38 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-04-23 13:33 IO speed limited by size of IO request (for RBD driver) Sylvain Munaut
2013-04-23 13:41 ` Steven Haigh
2013-04-23 14:06 ` Roger Pau Monné
2013-04-23 14:15 ` Sylvain Munaut
2013-04-25 13:00 ` Sylvain Munaut
[not found] ` <51769B9D.4000708@crc.id.au>
[not found] ` <51769CFD.7020907@citrix.com>
[not found] ` <51769E1E.6040902@crc.id.au>
[not found] ` <5176A19A.2010802@citrix.com>
[not found] ` <5176A440.8040303@crc.id.au>
[not found] ` <5176A520.5030503@citrix.com>
[not found] ` <5176A61F.6050607@crc.id.au>
[not found] ` <5176A6DD.5000404@citrix.com>
[not found] ` <5176AFF9.4020003@crc.id.au>
[not found] ` <5176B237.8020803@citrix.com>
[not found] ` <5176C073.3050409@crc.id.au>
[not found] ` <5176CF56.8000505@citrix.com>
[not found] ` <5176DB88.1070200@crc.id.au>
[not found] ` <517A89DA.3030804@citrix.com>
2013-04-26 14:16 ` Steven Haigh
2013-04-27 1:57 ` Steven Haigh
2013-04-27 7:06 ` Roger Pau Monné
2013-04-27 7:51 ` Steven Haigh
2013-04-27 8:35 ` Steven Haigh
2013-04-29 8:38 ` Roger Pau Monné [this message]
2013-04-29 19:26 ` Steven Haigh
2013-04-29 19:47 ` Steven Haigh
2013-04-30 10:07 ` Felipe Franciosi
2013-04-30 10:38 ` Steven Haigh
2013-05-08 8:20 ` Steven Haigh
2013-05-08 8:33 ` Roger Pau Monné
2013-05-08 8:47 ` Steven Haigh
2013-05-08 10:32 ` Steven Haigh
2013-05-08 10:45 ` Roger Pau Monné
2013-05-08 11:14 ` Felipe Franciosi
2013-05-22 20:13 ` Konrad Rzeszutek Wilk
2013-05-23 7:22 ` Felipe Franciosi
2013-05-24 14:29 ` Konrad Rzeszutek Wilk
2013-05-08 12:56 ` Steven Haigh
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=517E3195.8090204@citrix.com \
--to=roger.pau@citrix.com \
--cc=felipe.franciosi@citrix.com \
--cc=netwiz@crc.id.au \
--cc=xen-devel@lists.xen.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).