From: Yann Dupont <Yann.Dupont@univ-nantes.fr>
To: Stefan Priebe <s.priebe@profihost.ag>
Cc: Mark Nelson <mark.nelson@inktank.com>, ceph-devel@vger.kernel.org
Subject: Re: poor OSD performance using kernel 3.4
Date: Tue, 29 May 2012 23:31:12 +0200 [thread overview]
Message-ID: <4FC54020.8040603@univ-nantes.fr> (raw)
In-Reply-To: <4FC53AC4.5020600@profihost.ag>
Le 29/05/2012 23:08, Stefan Priebe a écrit :
> Am 29.05.2012 19:50, schrieb Mark Nelson:
>> I did some quick tests on a couple of nodes I had laying around this
>> morning.
>
> I just noticed that i get a constant rate of 40MB/s while using 1
> thread. When i use two thread or more i get drop to 0MB/s and crazy
> jumping values.
>
> ~# rados -p rbd bench 90 write -t 1
> Maintaining 1 concurrent writes of 4194304 bytes for at least 90 seconds.
> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> 0 0 0 0 0 0 - 0
> 1 1 10 9 35.994 36 0.100147 0.101133
> 2 1 20 19 37.9931 40 0.096893 0.100719
> 3 1 31 30 39.9921 44 0.09784 0.0999607
> 4 1 41 40 39.9929 40 0.099156 0.0999003
> 5 1 51 50 39.9932 40 0.098239 0.0996518
> 6 1 61 60 39.9932 40 0.098682 0.0994851
> 7 1 71 70 39.9933 40 0.094397 0.099184
> 8 1 81 80 39.9931 40 0.099823 0.0993327
> 9 1 91 90 39.9931 40 0.101013 0.0992236
> 10 1 101 100 39.993 40 0.098277 0.099237
>
>
not here :
on data :
root@label5:~# rados -p data bench 20 write -t 1
Maintaining 1 concurrent writes of 4194304 bytes for at least 20 seconds.
sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
0 0 0 0 0 0 - 0
1 1 15 14 55.9837 56 0.096813 0.0677311
2 1 33 32 63.9852 72 0.088802 0.0612602
3 1 51 50 66.6529 72 0.056883 0.0594909
4 1 60 59 58.989 36 0.046377 0.0577145
5 1 60 59 47.1916 0 - 0.0577145
6 1 79 78 51.9911 38 0.041831 0.0768918
7 1 98 97 55.419 76 0.050436 0.0718439
8 1 101 100 49.9919 12 0.043673 0.0712079
9 1 101 100 44.4375 0 - 0.0712079
10 1 115 114 45.5929 28 0.043768 0.0876947
11 1 134 133 48.356 76 0.052382 0.0826428
12 1 154 153 50.9919 80 0.042077 0.0783619
13 1 175 174 53.5299 84 0.053474 0.0745956
14 1 194 193 55.1339 76 0.049631 0.0724711
15 1 211 210 55.991 68 0.052683 0.0712887
16 1 232 231 57.7407 84 0.044341 0.0692121
17 1 249 248 58.3436 68 0.053707 0.0684414
18 1 258 257 57.102 36 0.086088 0.0680656
19 1 267 266 55.9911 36 0.050902 0.0713341
min lat: 0.033395 max lat: 2.14757 avg lat: 0.0703545
sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
20 1 285 284 56.7909 72 0.047755 0.0703545
Total time run: 20.066134
Total writes made: 286
Write size: 4194304
Bandwidth (MB/sec): 57.011
on rbd :
Maintaining 1 concurrent writes of 4194304 bytes for at least 20 seconds.
sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
0 1 1 0 0 0 - 0
1 1 18 17 67.9801 68 0.065869 0.0587313
2 1 35 34 67.9842 68 0.056982 0.0580468
3 1 55 54 71.9848 80 0.050305 0.0554721
4 1 72 71 70.9858 68 0.039387 0.0561269
5 1 91 90 71.986 76 0.055236 0.0554057
6 1 109 108 71.9864 72 0.069547 0.0554112
7 1 126 125 71.4154 68 0.049234 0.0556564
8 1 146 145 72.4868 80 0.052302 0.0551064
9 1 165 164 72.8758 76 0.0533 0.0548858
10 1 184 183 73.187 76 0.041342 0.0543598
11 1 202 201 73.078 72 0.048963 0.0544978
12 1 218 217 72.3207 64 0.071926 0.0549402
13 1 236 235 72.2951 72 0.055804 0.0551936
14 1 254 253 72.2731 72 0.058315 0.0552612
15 1 272 271 72.2541 72 0.047687 0.0552036
16 1 290 289 72.2375 72 0.059162 0.055275
17 1 308 307 72.2229 72 0.051991 0.0553467
18 1 327 326 72.432 76 0.053271 0.0552114
19 1 346 345 72.6192 76 0.058125 0.0550658
min lat: 0.036202 max lat: 0.113077 avg lat: 0.0547502
sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
20 1 366 365 72.9874 80 0.036246 0.0547502
Total time run: 20.086555
Total writes made: 367
Write size: 4194304
Bandwidth (MB/sec): 73.084
>
> # rados -p rbd bench 90 write -t 2
> Maintaining 2 concurrent writes of 4194304 bytes for at least 90 seconds.
> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> 0 0 0 0 0 0 - 0
> 1 2 15 13 51.9888 52 0.0956 0.115315
> 2 2 22 20 39.9928 28 0.120065 0.193125
> 3 2 41 39 51.9917 76 0.09557 0.15246
> 4 2 58 56 55.9912 68 0.09875 0.137688
> 5 2 67 65 51.992 36 0.111211 0.139465
> 6 2 85 83 55.3251 72 0.136967 0.143079
> 7 2 101 99 56.5625 64 0.098664 0.136263
> 8 2 101 99 49.4919 0 - 0.136263
> 9 2 112 110 48.8808 22 0.099479 0.160563
>
> Stefan
pool rbd stays consistent here, no matter how much thread involved. The
max speed with my setup is around 16~24 threads, and it's quite effective.
on the contrary, pool data is jumping up & down, no matter how much
thread involved :)
Maybe this is because journal is too tight ? Or because 2 of the 8 nodes
have slower disks ?
I may be able to retest thursday, my two last osd should have faster &
larger disks.
Cheers,
--
Yann Dupont - Service IRTS, DSI Université de Nantes
Tel : 02.53.48.49.20 - Mail/Jabber : Yann.Dupont@univ-nantes.fr
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2012-05-29 21:31 UTC|newest]
Thread overview: 73+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-05-24 14:10 poor OSD performance using kernel 3.4 Stefan Priebe - Profihost AG
2012-05-24 14:57 ` Mark Nelson
[not found] ` <CAJCPpW+SKnnVUaDEAsCkKyZwMVrHCRJF2C8zqB4eORgwW5p=1Q@mail.gmail.com>
[not found] ` <4FBE7ABC.5020502@profihost.ag>
2012-05-24 18:53 ` Mark Nelson
2012-05-24 19:05 ` Stefan Priebe
2012-05-25 1:53 ` Mark Nelson
2012-05-25 8:19 ` Stefan Priebe - Profihost AG
2012-05-25 11:31 ` Stefan Priebe - Profihost AG
2012-05-25 12:10 ` Stefan Priebe - Profihost AG
2012-05-25 15:47 ` Alexandre DERUMIER
2012-05-27 9:11 ` Stefan Priebe - Profihost AG
2012-05-27 11:33 ` Alexandre DERUMIER
2012-05-27 18:57 ` Stefan Priebe
2012-05-28 5:37 ` Alexandre DERUMIER
2012-05-28 6:25 ` Stefan Priebe
2012-05-28 6:52 ` Alexandre DERUMIER
2012-05-28 19:48 ` Stefan Priebe
2012-05-29 3:54 ` Alexandre DERUMIER
2012-05-29 8:22 ` Stefan Priebe - Profihost AG
2012-05-29 13:01 ` Alexandre DERUMIER
2012-05-29 14:18 ` Stefan Priebe - Profihost AG
2012-05-29 9:46 ` Stefan Priebe - Profihost AG
2012-05-29 13:39 ` Yann Dupont
2012-05-29 14:43 ` Stefan Priebe - Profihost AG
2012-05-29 17:50 ` Mark Nelson
2012-05-29 19:50 ` Yann Dupont
2012-05-29 21:04 ` Stefan Priebe
2012-05-29 21:08 ` Stefan Priebe
2012-05-29 21:31 ` Yann Dupont [this message]
2012-05-29 21:34 ` Stefan Priebe
2012-05-29 21:45 ` Yann Dupont
2012-05-30 6:29 ` Stefan Priebe - Profihost AG
2012-05-29 21:41 ` Mark Nelson
2012-05-30 6:22 ` Stefan Priebe - Profihost AG
2012-05-30 7:20 ` building test cluster : missing /etc/ceph/client.admin.keyring, need help Alexandre DERUMIER
2012-05-30 7:25 ` Stefan Priebe - Profihost AG
2012-05-30 7:33 ` Alexandre DERUMIER
2012-05-30 7:47 ` Alexandre DERUMIER
2012-05-29 22:25 ` poor OSD performance using kernel 3.4 Mark Nelson
2012-05-30 6:33 ` Stefan Priebe - Profihost AG
[not found] ` <CADdPHGs9dpSh9Oyu+5yDhyYU=Et_-zF5MuYybBuuAN5DgR433A@mail.gmail.com>
2012-05-30 7:16 ` Stefan Priebe - Profihost AG
[not found] ` <CADdPHGuiJqZUCK-0qR_CrOo6GRhkjaCdkOhJ2boq3zD0_voTsA@mail.gmail.com>
2012-05-30 11:04 ` Stefan Priebe - Profihost AG
[not found] ` <CADdPHGuLAL5+hkzq0tigqu355DvPxkhE5sxBhOVZPj=EzDSVtA@mail.gmail.com>
2012-05-30 11:25 ` Stefan Priebe - Profihost AG
2012-05-30 12:17 ` Mark Nelson
2012-05-30 12:41 ` Stefan Priebe - Profihost AG
[not found] ` <CADdPHGsmr8Ht1pTWH1Oe8=NmAyM81SSdH+c_GV89D8ntfyUmgA@mail.gmail.com>
2012-05-30 13:19 ` Stefan Priebe - Profihost AG
[not found] ` <CADdPHGvxCmuViy+0==Vkdz_QjC1K+kD5kD1m7+0tYM2YDTtJbw@mail.gmail.com>
2012-05-30 13:54 ` Stefan Priebe - Profihost AG
[not found] ` <4FC63381.6090300@inktank.com>
2012-05-30 14:53 ` Stefan Priebe
2012-05-30 14:56 ` Mark Nelson
2012-05-30 18:26 ` Stefan Priebe
2012-05-30 19:41 ` Mark Nelson
2012-05-30 13:27 ` Mark Nelson
2012-05-30 13:51 ` Stefan Priebe - Profihost AG
2012-05-30 14:16 ` Mark Nelson
2012-05-30 18:42 ` Stefan Priebe
[not found] ` <CADdPHGuxa7TAyqXcXehb9WgKgkHwkybYTrj2oue_PKsiF+oR3A@mail.gmail.com>
2012-05-30 21:10 ` Stefan Priebe
[not found] ` <CADdPHGutEwoDc=Kcrqcx2ZMO=dqhuoT5iLoP-WxqD+e5ZUmBRA@mail.gmail.com>
2012-05-31 7:10 ` poor OSD performance using kernel 3.4 => problem found Stefan Priebe - Profihost AG
2012-05-31 7:30 ` Yehuda Sadeh
[not found] ` <CADdPHGtz9Jq624DMO6Dve2AcJ9vrnFHbyqRa+qheA+0-y4k++g@mail.gmail.com>
2012-05-31 12:31 ` Mark Nelson
2012-05-31 12:33 ` Stefan Priebe - Profihost AG
2012-05-31 13:21 ` Yann Dupont
2012-05-31 13:37 ` Stefan Priebe - Profihost AG
2012-05-31 13:45 ` Yann Dupont
2012-05-31 14:42 ` Yann Dupont
2012-05-31 15:32 ` Mark Nelson
2012-05-31 15:43 ` Yann Dupont
2012-05-31 16:14 ` Mark Nelson
2012-05-31 16:29 ` Sage Weil
2012-05-31 16:37 ` Yann Dupont
[not found] ` <CADdPHGv0YjxDQFnZML-55jDj7XxHxaxUZ_FeQ=ReKK6Rs7NNhw@mail.gmail.com>
2012-05-31 8:04 ` Stefan Priebe - Profihost AG
2012-05-31 8:09 ` Stefan Majer
2012-05-31 11:34 ` Stefan Priebe - Profihost AG
2012-05-31 12:18 ` Stefan Priebe - Profihost AG
2012-05-30 11:51 ` poor OSD performance using kernel 3.4 Mark Nelson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4FC54020.8040603@univ-nantes.fr \
--to=yann.dupont@univ-nantes.fr \
--cc=ceph-devel@vger.kernel.org \
--cc=mark.nelson@inktank.com \
--cc=s.priebe@profihost.ag \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox