Re: IO speed limited by size of IO request (for RBD driver)

[Steven Haigh <netwiz@crc.id.au>]

On 27/04/2013 5:51 PM, Steven Haigh wrote:
> On 27/04/2013 5:06 PM, Roger Pau Monné wrote:
>> On 27/04/13 03:57, Steven Haigh wrote:
>>> On 27/04/2013 12:16 AM, Steven Haigh wrote:
>>>> On 27/04/2013 12:06 AM, Roger Pau Monné wrote:
>>>>> On 23/04/13 21:05, Steven Haigh wrote:
>>>>>> Sorry - resending this to Felipe as well - as I started talking to
>>>>>> him
>>>>>> directly previously.
>>>>>>
>>>>>> Felipe, to bring you up to date, I've copied over the blkback
>>>>>> files from
>>>>>> Rogers indirect kernel over the vanilla 3.8.8 kernel files, built and
>>>>>> tested. Results below:
>>>>>>
>>>>
>>>> Bringing this into context in a nutshell - results showed about 5MB/sec
>>>> improvement when using buffered disk access - totalling ~57MB/sec write
>>>> speed vs ~98MB/sec when using the oflag=direct flag to dd.
>>>>
>>>> When talking about back porting a few indirect patches to mainline
>>>> blkback (3.8.8 atm):
>>>>>> On 24/04/2013 4:13 AM, Roger Pau Monné wrote:
>>>>>>> I think it requires a non-trivial amount of work, what you could do
>>>>>>> as a
>>>>>>> test is directly replace the affected files with the ones in my
>>>>>>> tree, it
>>>>>>> is not optimal, but I don't think it's going to cause problems,
>>>>>>> and you
>>>>>>> could at least see if indirect descriptors solve your problem.
>>>>>>
>>>>>> Ok, I copied across those files, built, packaged and installed
>>>>>> them on
>>>>>> my Dom0. Good news is that its a little quicker, bad news is not by
>>>>>> much.
>>>>>
>>>>> Could you try increasing xen_blkif_max_segments variable in
>>>>> xen-blkfront.c to 64 or 128? It is set to 32 by default. You will only
>>>>> need to recompile the DomU kernel after this, the Dom0 is able to
>>>>> support up to 256 indirect segments.
>>>>
>>>> I'll have to look at this. All DomU's are Scientific Linux 6.4
>>>> systems -
>>>> so essentially RHEL6.4 and so on. I haven't built a RH kernel as yet -
>>>> so I'll have to look at what is involved. It might be as simple as
>>>> rebuilding a normal SRPM.
>>>
>>> Ok, I've had a look at the RH xen-blkfront.c - and I can't see any
>>> definition of xen_blkif_max_segments - or anything close. I've attached
>>> the version used in the EL6 kernel from the kernel-2.6.32-358.6.1.el6
>>> srpm.
>>>
>>> Any ideas on where to go from here?
>>
>> I thought you were using the 3.8.x kernel inside the DomU also, if you
>> are not using it, then it's normal that there's no speed difference, you
>> have a Dom0 kernel that supports indirect descriptors, but your DomU
>> doesn't. You must use a kernel that supports indirect descriptors in
>> both Dom0 and DomU in order to make use of this feature.
>
> Ahhh - sorry - I should have been clearer - The Dom0 is kernel 3.8.x
> (3.8.8 right now) - however the DomUs are all stock EL6 kernels.
>
> Hmmmm - I believe the kernel I build for Dom0 *should* work as a DomU.
> I'll do some more experimentation and see if I can get it working
> properly as a DomU kernel.

Ok - now for testing the basics.

Same tests using vanilla 3.8.8 kernel:
# dd if=/dev/zero of=output.zero bs=1M count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 37.1206 s, 57.9 MB/s

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
            0.00    0.00    2.65    0.00    0.22   97.13

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sdf             395.81  2201.32   27.59  156.95     1.65     9.21 
120.60     1.13    6.12   1.19  22.05
sde             404.86  2208.83   28.04  157.40     1.69     9.24 
120.77     1.32    7.15   1.31  24.24
sdc             435.54  2174.83   30.68  155.63     1.82     9.10 
120.09     0.97    5.20   1.11  20.64
sdd             388.96  2177.26   26.71  155.41     1.62     9.11 
120.74     1.10    6.01   1.30  23.60
md2               0.00     0.00    0.00  537.31     0.00    17.59 
67.05     0.00    0.00   0.00   0.00

# dd if=/dev/zero of=output.zero bs=1M count=2048 oflag=direct
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 25.3928 s, 84.6 MB/s

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
            0.22    0.00   15.74    0.00    0.22   83.81

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sdf             336.81 12659.65   10.86  232.59     1.36    50.36 
435.06     1.00    4.09   2.51  61.06
sde            1684.04 11000.22   54.32  189.14     6.79    43.71 
424.80     1.45    5.95   3.50  85.28
sdc             144.35 11177.61    4.66  238.80     0.58    44.60 
380.04     0.41    1.70   1.07  26.08
sdd              20.62 12876.50    0.67  242.79     0.08    51.25 
431.80     0.45    1.84   1.15  27.92
md2               0.00     0.00    0.00 2680.71     0.00    86.47 
66.06     0.00    0.00   0.00   0.00

Installed and rebooted into the patched version I build by just copying 
the blkback files across to the 3.8.8 tree and building:

# dd if=/dev/zero of=output.zero bs=1M count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 45.2376 s, 47.5 MB/s

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
            0.00    0.00    1.35    0.00    0.45   98.19

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sdd            1340.80  5806.80  158.20  674.40     5.83    25.27 
76.51     6.00    7.16   0.80  66.90
sde            1334.60  5894.00  160.80  686.40     5.86    25.71 
76.32     6.87    8.11   0.87  73.52
sdc            1330.80  5858.20  158.00  682.40     5.86    25.60 
76.67     5.71    6.81   0.77  64.84
sdf            1341.00  5848.80  157.00  681.20     5.83    25.49 
76.53     6.23    7.38   0.85  70.92
md2               0.00     0.00    0.00 1431.40     0.00    46.83 
67.01     0.00    0.00   0.00   0.00

# dd if=/dev/zero of=output.zero bs=1M count=2048 oflag=direct
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 38.9052 s, 55.2 MB/s

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
            0.00    0.00    5.27    0.00    0.32   94.41

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sdd             493.20  8481.60   36.80  335.80     2.07    34.45 
200.73     1.14    3.07   0.97  36.32
sde            1371.60  7380.20   83.80  304.80     5.66    30.08 
188.34     2.20    5.65   1.94  75.38
sdc             540.20  7556.80   56.00  326.20     2.33    30.80 
177.52     1.49    3.90   1.26  48.02
sdf             734.20  8286.60   64.40  326.20     3.12    33.67 
192.92     1.66    4.24   1.45  56.66
md2               0.00     0.00    0.00 1835.20     0.00    59.20 
66.06     0.00    0.00   0.00   0.00

That is with the same kernel running on both Dom0 and DomU.

In the dmesg of the DomU, I see the following:
blkfront: xvdb: flush diskcache: enabled using persistent grants

-- 
Steven Haigh

Email: netwiz@crc.id.au
Web: https://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897
Fax: (03) 8338 0299