Re: poor OSD performance using kernel 3.4 => problem found

[Yann Dupont <Yann.Dupont@univ-nantes.fr>]

On 31/05/2012 15:37, Stefan Priebe - Profihost AG wrote:
> Am 31.05.2012 15:21, schrieb Yann Dupont:
>> On 31/05/2012 09:30, Yehuda Sadeh wrote:
>>> On Thu, May 31, 2012 at 12:10 AM, Stefan Priebe - Profihost AG
>>> <s.priebe@profihost.ag> wrote:
>> But very strangely it's now rbd that isn't stable ?!
>>
>> root@label5:~# rados -p rbd bench 20 write -t 16
>> Maintaining 16 concurrent writes of 4194304 bytes for at least 20
>> seconds.
>> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
>> 0 0 0 0 0 0 - 0
>> 1 16 155 139 555.87 556 0.046232 0.109021
>> 2 16 250 234 467.923 380 0.046793 0.0985316
>> 3 16 250 234 311.955 0 - 0.0985316
>> 4 16 250 234 233.965 0 - 0.0985316
>> 5 16 250 234 187.173 0 - 0.0985316
>> 6 16 266 250 166.645 16 0.038083 0.175697
>> 7 16 266 250 142.839 0 - 0.175697
>> 8 16 441 425 212.475 350 0.05512 0.298391
>> 9 16 476 460 204.422 140 0.04372 0.280483
>> 10 16 531 515 205.976 220 0.125076 0.309449
>> 11 16 734 718 261.06 812 0.127582 0.244134
>> 12 16 795 779 259.637 244 0.065158 0.234156
>> 13 16 818 802 246.742 92 0.054514 0.241704
>> 14 16 830 814 232.546 48 0.044386 0.239006
>> 15 16 837 821 218.909 28 3.41523 0.267521
>> 16 16 1043 1027 256.721 824 0.04898 0.248212
>> 17 16 1147 1131 266.088 416 0.048591 0.232725
>> 18 16 1147 1131 251.305 0 - 0.232725
>> 19 16 1202 1186 249.657 110 0.081777 0.25501
>> min lat: 0.033773 max lat: 5.92059 avg lat: 0.245711
>> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
>> 20 16 1296 1280 255.97 376 0.053797 0.245711
>> 21 9 1297 1288 245.305 32 0.708133 0.248248
>> 22 9 1297 1288 234.155 0 - 0.248248
>> 23 9 1297 1288 223.975 0 - 0.248248
>> 24 9 1297 1288 214.643 0 - 0.248248
>> 25 9 1297 1288 206.057 0 - 0.248248
>> 26 9 1297 1288 198.131 0 - 0.248248
>> Total time run: 26.829870
>> Total writes made: 1297
>> Write size: 4194304
>> Bandwidth (MB/sec): 193.367
>>
>> Average Latency: 0.295922
>> Max latency: 7.36701
>> Min latency: 0.033773
>>
>>
>> Strange. I'm wondering if this has something to do with cache (that is,
>> operation I could have done before on nodes, as all my nodes are just
>> freshly rebooted).
>
> Please test setting these values on all OSDs and Clients:
> sysctl -w net.ipv4.tcp_rmem="4096        87380   514873"
> sysctl -w net.ipv4.tcp_wmem="4096        16384   514873"
>
> Stefan

same. stable for pool data (845 MB/s average), jumping with rbd (229 
average, with a max latency of 6).

I'm with latest linus git kernel
(commit af56e0aa35f3ae2a4c1a6d1000702df1dd78cb76) , and I based on the 
fact that the patch was reversed on it.

I can try with plain 3.4.0 with 'culprit patch' manually reversed.

what puzzles me is that this morning, with 3.4.0 it was rbd that was 
stable, and now I have the exact contrary.

I'll begin to reboot with old 3.4.0 kernel to see if things are 
reproductible.

Cheers,
-- 
Yann Dupont - Service IRTS, DSI Université de Nantes
Tel : 02.53.48.49.20 - Mail/Jabber : Yann.Dupont@univ-nantes.fr


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html