Re: IO speed limited by size of IO request (for RBD driver)

["Roger Pau Monné" <roger.pau@citrix.com>]

On 23/04/13 15:41, Steven Haigh wrote:
> On 23/04/2013 11:33 PM, Sylvain Munaut wrote:
>> Hi,
>>
>>
>> I was observing a pretty severe performance impact when using Xen VM
>> with RBD (Ceph) backed storage, especially when doing large sequential
>> access.
>>
>> And I think I finally found a major cause for it: even large user
>> space requests seem to be split into small requests of 11 * 4096
>> bytes. ( 44k )
>>
>> This is caused by :
>>
>>          blk_queue_max_segments(rq, BLKIF_MAX_SEGMENTS_PER_REQUEST);
>>          blk_queue_max_segment_size(rq, PAGE_SIZE);
>>
>> What's the impact of modifying those ?
>> I've seen some justification for the BLKIF_MAX_SEGMENTS_PER_REQUEST
>> limit, but why limit segment size to page_size ?
> 
> I'm seeing the same as you - see the thread in the archives over the 
> last few weeks - subject "RE: Xen disk write slowness in kernel 3.8.x"
> 
> I get ~50MB/sec max write speeds due to probably the same problem.

When using Ceph, are you using the Linux kernel backend (blkback), Qemu
or blktap?

I've been working on expanding the number of segments that a request can
hold, and the patches just went upstream for the next Linux kernel
(3.10), you might want to test them, they can be found in the following
git repo:

git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git

branch for-jens-3.10

You will need to use them in both DomU and Dom0 in order to get more
segments in a request.