From: Yann Dupont <Yann.Dupont@univ-nantes.fr>
To: Stefan Priebe <s.priebe@profihost.ag>
Cc: Mark Nelson <mark.nelson@inktank.com>, ceph-devel@vger.kernel.org
Subject: Re: poor OSD performance using kernel 3.4
Date: Tue, 29 May 2012 23:31:12 +0200 [thread overview]
Message-ID: <4FC54020.8040603@univ-nantes.fr> (raw)
In-Reply-To: <4FC53AC4.5020600@profihost.ag>
Le 29/05/2012 23:08, Stefan Priebe a écrit :
> Am 29.05.2012 19:50, schrieb Mark Nelson:
>> I did some quick tests on a couple of nodes I had laying around this
>> morning.
>
> I just noticed that i get a constant rate of 40MB/s while using 1
> thread. When i use two thread or more i get drop to 0MB/s and crazy
> jumping values.
>
> ~# rados -p rbd bench 90 write -t 1
> Maintaining 1 concurrent writes of 4194304 bytes for at least 90 seconds.
> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> 0 0 0 0 0 0 - 0
> 1 1 10 9 35.994 36 0.100147 0.101133
> 2 1 20 19 37.9931 40 0.096893 0.100719
> 3 1 31 30 39.9921 44 0.09784 0.0999607
> 4 1 41 40 39.9929 40 0.099156 0.0999003
> 5 1 51 50 39.9932 40 0.098239 0.0996518
> 6 1 61 60 39.9932 40 0.098682 0.0994851
> 7 1 71 70 39.9933 40 0.094397 0.099184
> 8 1 81 80 39.9931 40 0.099823 0.0993327
> 9 1 91 90 39.9931 40 0.101013 0.0992236
> 10 1 101 100 39.993 40 0.098277 0.099237
>
>
not here :
on data :
root@label5:~# rados -p data bench 20 write -t 1
Maintaining 1 concurrent writes of 4194304 bytes for at least 20 seconds.
sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
0 0 0 0 0 0 - 0
1 1 15 14 55.9837 56 0.096813 0.0677311
2 1 33 32 63.9852 72 0.088802 0.0612602
3 1 51 50 66.6529 72 0.056883 0.0594909
4 1 60 59 58.989 36 0.046377 0.0577145
5 1 60 59 47.1916 0 - 0.0577145
6 1 79 78 51.9911 38 0.041831 0.0768918
7 1 98 97 55.419 76 0.050436 0.0718439
8 1 101 100 49.9919 12 0.043673 0.0712079
9 1 101 100 44.4375 0 - 0.0712079
10 1 115 114 45.5929 28 0.043768 0.0876947
11 1 134 133 48.356 76 0.052382 0.0826428
12 1 154 153 50.9919 80 0.042077 0.0783619
13 1 175 174 53.5299 84 0.053474 0.0745956
14 1 194 193 55.1339 76 0.049631 0.0724711
15 1 211 210 55.991 68 0.052683 0.0712887
16 1 232 231 57.7407 84 0.044341 0.0692121
17 1 249 248 58.3436 68 0.053707 0.0684414
18 1 258 257 57.102 36 0.086088 0.0680656
19 1 267 266 55.9911 36 0.050902 0.0713341
min lat: 0.033395 max lat: 2.14757 avg lat: 0.0703545
sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
20 1 285 284 56.7909 72 0.047755 0.0703545
Total time run: 20.066134
Total writes made: 286
Write size: 4194304
Bandwidth (MB/sec): 57.011
on rbd :
Maintaining 1 concurrent writes of 4194304 bytes for at least 20 seconds.
sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
0 1 1 0 0 0 - 0
1 1 18 17 67.9801 68 0.065869 0.0587313
2 1 35 34 67.9842 68 0.056982 0.0580468
3 1 55 54 71.9848 80 0.050305 0.0554721
4 1 72 71 70.9858 68 0.039387 0.0561269
5 1 91 90 71.986 76 0.055236 0.0554057
6 1 109 108 71.9864 72 0.069547 0.0554112
7 1 126 125 71.4154 68 0.049234 0.0556564
8 1 146 145 72.4868 80 0.052302 0.0551064
9 1 165 164 72.8758 76 0.0533 0.0548858
10 1 184 183 73.187 76 0.041342 0.0543598
11 1 202 201 73.078 72 0.048963 0.0544978
12 1 218 217 72.3207 64 0.071926 0.0549402
13 1 236 235 72.2951 72 0.055804 0.0551936
14 1 254 253 72.2731 72 0.058315 0.0552612
15 1 272 271 72.2541 72 0.047687 0.0552036
16 1 290 289 72.2375 72 0.059162 0.055275
17 1 308 307 72.2229 72 0.051991 0.0553467
18 1 327 326 72.432 76 0.053271 0.0552114
19 1 346 345 72.6192 76 0.058125 0.0550658
min lat: 0.036202 max lat: 0.113077 avg lat: 0.0547502
sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
20 1 366 365 72.9874 80 0.036246 0.0547502
Total time run: 20.086555
Total writes made: 367
Write size: 4194304
Bandwidth (MB/sec): 73.084
>
> # rados -p rbd bench 90 write -t 2
> Maintaining 2 concurrent writes of 4194304 bytes for at least 90 seconds.
> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> 0 0 0 0 0 0 - 0
> 1 2 15 13 51.9888 52 0.0956 0.115315
> 2 2 22 20 39.9928 28 0.120065 0.193125
> 3 2 41 39 51.9917 76 0.09557 0.15246
> 4 2 58 56 55.9912 68 0.09875 0.137688
> 5 2 67 65 51.992 36 0.111211 0.139465
> 6 2 85 83 55.3251 72 0.136967 0.143079
> 7 2 101 99 56.5625 64 0.098664 0.136263
> 8 2 101 99 49.4919 0 - 0.136263
> 9 2 112 110 48.8808 22 0.099479 0.160563
>
> Stefan
pool rbd stays consistent here, no matter how much thread involved. The
max speed with my setup is around 16~24 threads, and it's quite effective.
on the contrary, pool data is jumping up & down, no matter how much
thread involved :)
Maybe this is because journal is too tight ? Or because 2 of the 8 nodes
have slower disks ?
I may be able to retest thursday, my two last osd should have faster &
larger disks.
Cheers,
--
Yann Dupont - Service IRTS, DSI Université de Nantes
Tel : 02.53.48.49.20 - Mail/Jabber : Yann.Dupont@univ-nantes.fr
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2012-05-29 21:31 UTC|newest]
Thread overview: 73+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-05-24 14:10 poor OSD performance using kernel 3.4 Stefan Priebe - Profihost AG
2012-05-24 14:57 ` Mark Nelson
[not found] ` <CAJCPpW+SKnnVUaDEAsCkKyZwMVrHCRJF2C8zqB4eORgwW5p=1Q@mail.gmail.com>
[not found] ` <4FBE7ABC.5020502@profihost.ag>
2012-05-24 18:53 ` Mark Nelson
2012-05-24 19:05 ` Stefan Priebe
2012-05-25 1:53 ` Mark Nelson
2012-05-25 8:19 ` Stefan Priebe - Profihost AG
2012-05-25 11:31 ` Stefan Priebe - Profihost AG
2012-05-25 12:10 ` Stefan Priebe - Profihost AG
2012-05-25 15:47 ` Alexandre DERUMIER
2012-05-27 9:11 ` Stefan Priebe - Profihost AG
2012-05-27 11:33 ` Alexandre DERUMIER
2012-05-27 18:57 ` Stefan Priebe
2012-05-28 5:37 ` Alexandre DERUMIER
2012-05-28 6:25 ` Stefan Priebe
2012-05-28 6:52 ` Alexandre DERUMIER
2012-05-28 19:48 ` Stefan Priebe
2012-05-29 3:54 ` Alexandre DERUMIER
2012-05-29 8:22 ` Stefan Priebe - Profihost AG
2012-05-29 13:01 ` Alexandre DERUMIER
2012-05-29 14:18 ` Stefan Priebe - Profihost AG
2012-05-29 9:46 ` Stefan Priebe - Profihost AG
2012-05-29 13:39 ` Yann Dupont
2012-05-29 14:43 ` Stefan Priebe - Profihost AG
2012-05-29 17:50 ` Mark Nelson
2012-05-29 19:50 ` Yann Dupont
2012-05-29 21:04 ` Stefan Priebe
2012-05-29 21:08 ` Stefan Priebe
2012-05-29 21:31 ` Yann Dupont [this message]
2012-05-29 21:34 ` Stefan Priebe
2012-05-29 21:45 ` Yann Dupont
2012-05-30 6:29 ` Stefan Priebe - Profihost AG
2012-05-29 21:41 ` Mark Nelson
2012-05-30 6:22 ` Stefan Priebe - Profihost AG
2012-05-30 7:20 ` building test cluster : missing /etc/ceph/client.admin.keyring, need help Alexandre DERUMIER
2012-05-30 7:25 ` Stefan Priebe - Profihost AG
2012-05-30 7:33 ` Alexandre DERUMIER
2012-05-30 7:47 ` Alexandre DERUMIER
2012-05-29 22:25 ` poor OSD performance using kernel 3.4 Mark Nelson
2012-05-30 6:33 ` Stefan Priebe - Profihost AG
[not found] ` <CADdPHGs9dpSh9Oyu+5yDhyYU=Et_-zF5MuYybBuuAN5DgR433A@mail.gmail.com>
2012-05-30 7:16 ` Stefan Priebe - Profihost AG
[not found] ` <CADdPHGuiJqZUCK-0qR_CrOo6GRhkjaCdkOhJ2boq3zD0_voTsA@mail.gmail.com>
2012-05-30 11:04 ` Stefan Priebe - Profihost AG
[not found] ` <CADdPHGuLAL5+hkzq0tigqu355DvPxkhE5sxBhOVZPj=EzDSVtA@mail.gmail.com>
2012-05-30 11:25 ` Stefan Priebe - Profihost AG
2012-05-30 12:17 ` Mark Nelson
2012-05-30 12:41 ` Stefan Priebe - Profihost AG
[not found] ` <CADdPHGsmr8Ht1pTWH1Oe8=NmAyM81SSdH+c_GV89D8ntfyUmgA@mail.gmail.com>
2012-05-30 13:19 ` Stefan Priebe - Profihost AG
[not found] ` <CADdPHGvxCmuViy+0==Vkdz_QjC1K+kD5kD1m7+0tYM2YDTtJbw@mail.gmail.com>
2012-05-30 13:54 ` Stefan Priebe - Profihost AG
[not found] ` <4FC63381.6090300@inktank.com>
2012-05-30 14:53 ` Stefan Priebe
2012-05-30 14:56 ` Mark Nelson
2012-05-30 18:26 ` Stefan Priebe
2012-05-30 19:41 ` Mark Nelson
2012-05-30 13:27 ` Mark Nelson
2012-05-30 13:51 ` Stefan Priebe - Profihost AG
2012-05-30 14:16 ` Mark Nelson
2012-05-30 18:42 ` Stefan Priebe
[not found] ` <CADdPHGuxa7TAyqXcXehb9WgKgkHwkybYTrj2oue_PKsiF+oR3A@mail.gmail.com>
2012-05-30 21:10 ` Stefan Priebe
[not found] ` <CADdPHGutEwoDc=Kcrqcx2ZMO=dqhuoT5iLoP-WxqD+e5ZUmBRA@mail.gmail.com>
2012-05-31 7:10 ` poor OSD performance using kernel 3.4 => problem found Stefan Priebe - Profihost AG
2012-05-31 7:30 ` Yehuda Sadeh
[not found] ` <CADdPHGtz9Jq624DMO6Dve2AcJ9vrnFHbyqRa+qheA+0-y4k++g@mail.gmail.com>
2012-05-31 12:31 ` Mark Nelson
2012-05-31 12:33 ` Stefan Priebe - Profihost AG
2012-05-31 13:21 ` Yann Dupont
2012-05-31 13:37 ` Stefan Priebe - Profihost AG
2012-05-31 13:45 ` Yann Dupont
2012-05-31 14:42 ` Yann Dupont
2012-05-31 15:32 ` Mark Nelson
2012-05-31 15:43 ` Yann Dupont
2012-05-31 16:14 ` Mark Nelson
2012-05-31 16:29 ` Sage Weil
2012-05-31 16:37 ` Yann Dupont
[not found] ` <CADdPHGv0YjxDQFnZML-55jDj7XxHxaxUZ_FeQ=ReKK6Rs7NNhw@mail.gmail.com>
2012-05-31 8:04 ` Stefan Priebe - Profihost AG
2012-05-31 8:09 ` Stefan Majer
2012-05-31 11:34 ` Stefan Priebe - Profihost AG
2012-05-31 12:18 ` Stefan Priebe - Profihost AG
2012-05-30 11:51 ` poor OSD performance using kernel 3.4 Mark Nelson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4FC54020.8040603@univ-nantes.fr \
--to=yann.dupont@univ-nantes.fr \
--cc=ceph-devel@vger.kernel.org \
--cc=mark.nelson@inktank.com \
--cc=s.priebe@profihost.ag \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.