Re: [RESEND] [PATCH] readahead:add blk_run_backing_dev

[Vladislav Bolkhovitin <vst@vlnb.net>]

Ronald Moesbergen, on 07/17/2009 06:15 PM wrote:
> 2009/7/16 Vladislav Bolkhovitin <vst@vlnb.net>:
>> Ronald Moesbergen, on 07/16/2009 11:32 AM wrote:
>>> 2009/7/15 Vladislav Bolkhovitin <vst@vlnb.net>:
>>>>> The drop with 64 max_sectors_kb on the client is a consequence of how
>>>>> CFQ
>>>>> is working. I can't find the exact code responsible for this, but from
>>>>> all
>>>>> signs, CFQ stops delaying requests if amount of outstanding requests
>>>>> exceeds
>>>>> some threshold, which is 2 or 3. With 64 max_sectors_kb and 5 SCST I/O
>>>>> threads this threshold is exceeded, so CFQ doesn't recover order of
>>>>> requests, hence the performance drop. With default 512 max_sectors_kb
>>>>> and
>>>>> 128K RA the server sees at max 2 requests at time.
>>>>>
>>>>> Ronald, can you perform the same tests with 1 and 2 SCST I/O threads,
>>>>> please?
>>> Ok. Should I still use the file-on-xfs testcase for this, or should I
>>> go back to using a regular block device?
>> Yes, please
>>
>>> The file-over-iscsi is quite
>>> uncommon I suppose, most people will export a block device over iscsi,
>>> not a file.
>> No, files are common. The main reason why people use direct block devices is
>> a not supported by anything believe that comparing with files they "have
>> less overhead", so "should be faster". But it isn't true and can be easily
>> checked.
>>
>>>> With context-RA patch, please, in those and future tests, since it should
>>>> make RA for cooperative threads much better.
>>>>
>>>>> You can limit amount of SCST I/O threads by num_threads parameter of
>>>>> scst_vdisk module.
>>> Ok, I'll try that and include the blk_run_backing_dev,
>>> readahead-context and io_context patches.
> 
> The results:
> 
> client kernel: 2.6.26-15lenny3 (debian)
> server kernel: 2.6.29.5 with readahead-context, blk_run_backing_dev
> and io_context
> 
> With one IO thread:
> 
> 5) client: default, server: default
> blocksize       R        R        R   R(avg,    R(std        R
>   (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
>  67108864  15.990   15.308   16.689   64.097    2.259    1.002
>  33554432  15.981   16.064   16.221   63.651    0.392    1.989
>  16777216  15.841   15.660   16.031   64.635    0.619    4.040
> 
> 6) client: default, server: 64 max_sectors_kb, RA default
> blocksize       R        R        R   R(avg,    R(std        R
>   (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
>  67108864  16.035   16.024   16.654   63.084    1.130    0.986
>  33554432  15.924   15.975   16.359   63.668    0.762    1.990
>  16777216  16.168   16.104   15.838   63.858    0.571    3.991
> 
> 7) client: default, server: default max_sectors_kb, RA 2MB
> blocksize       R        R        R   R(avg,    R(std        R
>   (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
>  67108864  14.895   16.142   15.998   65.398    2.379    1.022
>  33554432  16.753   16.169   16.067   62.729    1.146    1.960
>  16777216  16.866   15.912   16.099   62.892    1.570    3.931
> 
> 8) client: default, server: 64 max_sectors_kb, RA 2MB
> blocksize       R        R        R   R(avg,    R(std        R
>   (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
>  67108864  15.923   15.716   16.741   63.545    1.715    0.993
>  33554432  16.010   16.026   16.113   63.802    0.180    1.994
>  16777216  16.644   16.239   16.143   62.672    0.827    3.917
> 
> 9) client: 64 max_sectors_kb, default RA. server: 64 max_sectors_kb, RA 2MB
> blocksize       R        R        R   R(avg,    R(std        R
>   (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
>  67108864  15.753   15.882   15.482   65.207    0.697    1.019
>  33554432  15.670   16.268   15.669   64.548    1.134    2.017
>  16777216  15.746   15.519   16.411   64.471    1.516    4.029
> 
> 10) client: default max_sectors_kb, 2MB RA. server: 64 max_sectors_kb, RA 2MB
> blocksize       R        R        R   R(avg,    R(std        R
>   (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
>  67108864  13.639   14.360   13.654   73.795    1.758    1.153
>  33554432  13.584   13.938   14.538   73.095    2.035    2.284
>  16777216  13.617   13.510   13.803   75.060    0.665    4.691
> 
> 11) client: 64 max_sectors_kb, 2MB. RA server: 64 max_sectors_kb, RA 2MB
> blocksize       R        R        R   R(avg,    R(std        R
>   (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
>  67108864  13.428   13.541   14.144   74.760    1.690    1.168
>  33554432  13.707   13.352   13.462   75.821    0.827    2.369
>  16777216  14.380   13.504   13.675   73.975    1.991    4.623
> 
> With two threads:
> 5) client: default, server: default
> blocksize       R        R        R   R(avg,    R(std        R
>   (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
>  67108864  12.453   12.173   13.014   81.677    2.254    1.276
>  33554432  12.066   11.999   12.960   83.073    2.877    2.596
>  16777216  13.719   11.969   12.569   80.554    4.500    5.035
> 
> 6) client: default, server: 64 max_sectors_kb, RA default
> blocksize       R        R        R   R(avg,    R(std        R
>   (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
>  67108864  12.886   12.201   12.147   82.564    2.198    1.290
>  33554432  12.344   12.928   12.007   82.483    2.504    2.578
>  16777216  12.380   11.951   13.119   82.151    3.141    5.134
> 
> 7) client: default, server: default max_sectors_kb, RA 2MB
> blocksize       R        R        R   R(avg,    R(std        R
>   (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
>  67108864  12.824   13.485   13.534   77.148    1.913    1.205
>  33554432  12.084   13.752   12.111   81.251    4.800    2.539
>  16777216  12.658   13.035   11.196   83.640    5.612    5.227
> 
> 8) client: default, server: 64 max_sectors_kb, RA 2MB
> blocksize       R        R        R   R(avg,    R(std        R
>   (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
>  67108864  12.253   12.552   11.773   84.044    2.230    1.313
>  33554432  13.177   12.456   11.604   82.723    4.316    2.585
>  16777216  12.471   12.318   13.006   81.324    1.878    5.083
> 
> 9) client: 64 max_sectors_kb, default RA. server: 64 max_sectors_kb, RA 2MB
> blocksize       R        R        R   R(avg,    R(std        R
>   (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
>  67108864  14.409   13.311   14.278   73.238    2.624    1.144
>  33554432  14.665   14.260   14.080   71.455    1.211    2.233
>  16777216  14.179   14.810   14.640   70.438    1.303    4.402
> 
> 10) client: default max_sectors_kb, 2MB RA. server: 64 max_sectors_kb, RA 2MB
> blocksize       R        R        R   R(avg,    R(std        R
>   (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
>  67108864  13.401   14.107   13.549   74.860    1.642    1.170
>  33554432  14.575   13.221   14.428   72.894    3.236    2.278
>  16777216  13.771   14.227   13.594   73.887    1.408    4.618
> 
> 11) client: 64 max_sectors_kb, 2MB. RA server: 64 max_sectors_kb, RA 2MB
> blocksize       R        R        R   R(avg,    R(std        R
>   (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
>  67108864  10.286   12.272   10.245   94.317    7.690    1.474
>  33554432  10.241   10.415   13.374   91.624   10.670    2.863
>  16777216  10.499   10.224   10.792   97.526    2.151    6.095
> 
> The last result comes close to 100MB/s!

Good! Although I expected maximum with a single thread.

Can you do the same set of tests with deadline scheduler on the server?

Thanks,
Vlad