Re: IO speed limited by size of IO request (for RBD driver)

["Roger Pau Monné" <roger.pau@citrix.com>]

On 27/04/13 03:57, Steven Haigh wrote:
> On 27/04/2013 12:16 AM, Steven Haigh wrote:
>> On 27/04/2013 12:06 AM, Roger Pau Monné wrote:
>>> On 23/04/13 21:05, Steven Haigh wrote:
>>>> Sorry - resending this to Felipe as well - as I started talking to him
>>>> directly previously.
>>>>
>>>> Felipe, to bring you up to date, I've copied over the blkback files from
>>>> Rogers indirect kernel over the vanilla 3.8.8 kernel files, built and
>>>> tested. Results below:
>>>>
>>
>> Bringing this into context in a nutshell - results showed about 5MB/sec
>> improvement when using buffered disk access - totalling ~57MB/sec write
>> speed vs ~98MB/sec when using the oflag=direct flag to dd.
>>
>> When talking about back porting a few indirect patches to mainline
>> blkback (3.8.8 atm):
>>>> On 24/04/2013 4:13 AM, Roger Pau Monné wrote:
>>>>> I think it requires a non-trivial amount of work, what you could do
>>>>> as a
>>>>> test is directly replace the affected files with the ones in my
>>>>> tree, it
>>>>> is not optimal, but I don't think it's going to cause problems, and you
>>>>> could at least see if indirect descriptors solve your problem.
>>>>
>>>> Ok, I copied across those files, built, packaged and installed them on
>>>> my Dom0. Good news is that its a little quicker, bad news is not by
>>>> much.
>>>
>>> Could you try increasing xen_blkif_max_segments variable in
>>> xen-blkfront.c to 64 or 128? It is set to 32 by default. You will only
>>> need to recompile the DomU kernel after this, the Dom0 is able to
>>> support up to 256 indirect segments.
>>
>> I'll have to look at this. All DomU's are Scientific Linux 6.4 systems -
>> so essentially RHEL6.4 and so on. I haven't built a RH kernel as yet -
>> so I'll have to look at what is involved. It might be as simple as
>> rebuilding a normal SRPM.
> 
> Ok, I've had a look at the RH xen-blkfront.c - and I can't see any 
> definition of xen_blkif_max_segments - or anything close. I've attached 
> the version used in the EL6 kernel from the kernel-2.6.32-358.6.1.el6 srpm.
> 
> Any ideas on where to go from here?

I thought you were using the 3.8.x kernel inside the DomU also, if you
are not using it, then it's normal that there's no speed difference, you
have a Dom0 kernel that supports indirect descriptors, but your DomU
doesn't. You must use a kernel that supports indirect descriptors in
both Dom0 and DomU in order to make use of this feature.