Re: IO speed limited by size of IO request (for RBD driver)

[Steven Haigh <netwiz@crc.id.au>]

On 8/05/2013 6:33 PM, Roger Pau Monné wrote:
> On 08/05/13 10:20, Steven Haigh wrote:
>> On 30/04/2013 8:07 PM, Felipe Franciosi wrote:
>>> I noticed you copied your results from "dd", but I didn't see any conclusions drawn from experiment.
>>>
>>> Did I understand it wrong or now you have comparable performance on dom0 and domU when using DIRECT?
>>>
>>> domU:
>>> # dd if=/dev/zero of=output.zero bs=1M count=2048 oflag=direct
>>> 2048+0 records in
>>> 2048+0 records out
>>> 2147483648 bytes (2.1 GB) copied, 25.4705 s, 84.3 MB/s
>>>
>>> dom0:
>>> # dd if=/dev/zero of=output.zero bs=1M count=2048 oflag=direct
>>> 2048+0 records in
>>> 2048+0 records out
>>> 2147483648 bytes (2.1 GB) copied, 24.8914 s, 86.3 MB/s
>>>
>>>
>>> I think that if the performance differs when NOT using DIRECT, the issue must be related to the way your guest is flushing the cache. This must be generating a workload that doesn't perform well on Xen's PV protocol.
>>
>> Just wondering if there is any further input on this... While DIRECT
>> writes are as good as can be expected, NON-DIRECT writes in certain
>> cases (specifically with a mdadm raid in the Dom0) are affected by about
>> a 50% loss in throughput...
>>
>> The hard part is that this is the default mode of writing!
>
> As another test with indirect descriptors, could you change
> xen_blkif_max_segments in xen-blkfront.c to 128 (it is 32 by default),
> recompile the DomU kernel and see if that helps?

Ok, here we go.... compiled as 3.8.0-2 with the above change. 3.8.0-2 is 
running on both the Dom0 and DomU.

# dd if=/dev/zero of=output.zero bs=1M count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 22.1703 s, 96.9 MB/s

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
            0.34    0.00   17.10    0.00    0.23   82.33

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sdd             980.97 11936.47   53.11  429.78     4.00    48.77 
223.81    12.75   26.10   2.11 101.79
sdc             872.71 11957.87   45.98  435.67     3.55    49.30 
224.71    13.77   28.43   2.11 101.49
sde             949.26 11981.88   51.30  429.33     3.91    48.90 
225.03    21.29   43.91   2.27 109.08
sdf             915.52 11968.52   48.58  428.88     3.73    48.92 
225.84    21.44   44.68   2.27 108.56
md2               0.00     0.00    0.00 1155.61     0.00    97.51 
172.80     0.00    0.00   0.00   0.00

# dd if=/dev/zero of=output.zero bs=1M count=2048 oflag=direct
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 25.3708 s, 84.6 MB/s

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
            0.11    0.00   13.92    0.00    0.22   85.75

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sdd               0.00 13986.08    0.00  263.20     0.00    55.76 
433.87     0.43    1.63   1.07  28.27
sdc             202.10 13741.55    6.52  256.57     0.81    54.77 
432.65     0.50    1.88   1.25  32.78
sde              47.96 11437.57    1.55  261.77     0.19    45.79 
357.63     0.80    3.02   1.85  48.60
sdf            2233.37 11756.13   71.93  191.38     8.99    46.80 
433.90     1.49    5.66   3.27  86.15
md2               0.00     0.00    0.00  731.93     0.00    91.49 
256.00     0.00    0.00   0.00   0.00

Now this is pretty much exactly what I would expect the system to do.... 
~96MB/sec buffered, and 85MB/sec direct.

So - it turns out that xen_blkif_max_segments at 32 is a killer in the 
DomU. Now it makes me wonder what we can do about this in kernels that 
don't have your series of patches against it? And also about the backend 
stuff in 3.8.x etc?

-- 
Steven Haigh

Email: netwiz@crc.id.au
Web: https://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897
Fax: (03) 8338 0299