From: Yann Dupont <Yann.Dupont@univ-nantes.fr>
To: Yehuda Sadeh <yehuda@inktank.com>
Cc: Stefan Priebe - Profihost AG <s.priebe@profihost.ag>,
Stefan Majer <stefan.majer@gmail.com>,
Mark Nelson <mark.nelson@inktank.com>,
ceph-devel@vger.kernel.org
Subject: Re: poor OSD performance using kernel 3.4 => problem found
Date: Thu, 31 May 2012 15:21:09 +0200 [thread overview]
Message-ID: <4FC77045.6050907@univ-nantes.fr> (raw)
In-Reply-To: <CAC-hyiFjRFLVHYUKv8bGG0u8u2ZrHgP78U2Txt+3R7GGwtopZA@mail.gmail.com>
On 31/05/2012 09:30, Yehuda Sadeh wrote:
> On Thu, May 31, 2012 at 12:10 AM, Stefan Priebe - Profihost AG
> <s.priebe@profihost.ag> wrote:
>> Hi Marc, Hi Stefan,
>>
Hello, back today
Today, I upgraded my 2 last osd nodes with big storage, so now all my
nodes are equivalent.
Using 3.4.0 kernel, I still have good results with rbd pool, but jumping
values with data.
>> first thanks for all your help and time.
>>
>> I found the commit which results in this problem and it is TCP related
>> but i'm still wondering if the expected behaviour of this commit is
>> expected?
>
....
>>
> Yeah, this might have affected the tcp performance. Looking at the
> current linus tree this function looks more like it looked beforehand,
> so it was probable reverted this way or another!
>
> Yehuda
Well, I saw you probably found the culprit.
So tried the latest (this morning) git kernel.
Now data gives good results :
root@label5:~# rados -p data bench 20 write -t 16
Maintaining 16 concurrent writes of 4194304 bytes for at least 20 seconds.
sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
0 0 0 0 0 0 - 0
1 16 215 199 795.765 796 0.073769 0.0745517
2 16 430 414 827.833 860 0.060165 0.0753952
3 16 632 616 821.207 808 0.072241 0.0772463
4 16 838 822 821.883 824 0.129571 0.0768741
5 16 1039 1023 818.271 804 0.056867 0.077637
6 16 1254 1238 825.209 860 0.078801 0.0771122
7 16 1474 1458 833.023 880 0.062886 0.0764071
8 16 1669 1653 826.389 780 0.09632 0.0767323
9 16 1877 1861 827.003 832 0.083765 0.0770398
10 16 2087 2071 828.294 840 0.051437 0.076937
11 16 2309 2293 833.714 888 0.080584 0.0764829
12 16 2535 2519 839.563 904 0.078095 0.0759574
13 16 2762 2746 844.816 908 0.081323 0.0754571
14 16 2984 2968 847.889 888 0.076973 0.0752921
15 16 3203 3187 849.754 876 0.069877 0.0750613
16 16 3437 3421 855.138 936 0.046845 0.0746941
17 16 3655 3639 856.126 872 0.052258 0.0745157
18 16 3862 3846 854.559 828 0.061542 0.0746875
19 16 4085 4069 856.525 892 0.053889 0.0745582
min lat: 0.033007 max lat: 0.462951 avg lat: 0.0743988
sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
20 15 4308 4293 858.492 896 0.054176 0.0743988
Total time run: 20.103415
Total writes made: 4309
Write size: 4194304
Bandwidth (MB/sec): 857.367
Average Latency: 0.0746302
Max latency: 0.462951
Min latency: 0.033007
But very strangely it's now rbd that isn't stable ?!
root@label5:~# rados -p rbd bench 20 write -t 16
Maintaining 16 concurrent writes of 4194304 bytes for at least 20 seconds.
sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
0 0 0 0 0 0 - 0
1 16 155 139 555.87 556 0.046232 0.109021
2 16 250 234 467.923 380 0.046793 0.0985316
3 16 250 234 311.955 0 - 0.0985316
4 16 250 234 233.965 0 - 0.0985316
5 16 250 234 187.173 0 - 0.0985316
6 16 266 250 166.645 16 0.038083 0.175697
7 16 266 250 142.839 0 - 0.175697
8 16 441 425 212.475 350 0.05512 0.298391
9 16 476 460 204.422 140 0.04372 0.280483
10 16 531 515 205.976 220 0.125076 0.309449
11 16 734 718 261.06 812 0.127582 0.244134
12 16 795 779 259.637 244 0.065158 0.234156
13 16 818 802 246.742 92 0.054514 0.241704
14 16 830 814 232.546 48 0.044386 0.239006
15 16 837 821 218.909 28 3.41523 0.267521
16 16 1043 1027 256.721 824 0.04898 0.248212
17 16 1147 1131 266.088 416 0.048591 0.232725
18 16 1147 1131 251.305 0 - 0.232725
19 16 1202 1186 249.657 110 0.081777 0.25501
min lat: 0.033773 max lat: 5.92059 avg lat: 0.245711
sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
20 16 1296 1280 255.97 376 0.053797 0.245711
21 9 1297 1288 245.305 32 0.708133 0.248248
22 9 1297 1288 234.155 0 - 0.248248
23 9 1297 1288 223.975 0 - 0.248248
24 9 1297 1288 214.643 0 - 0.248248
25 9 1297 1288 206.057 0 - 0.248248
26 9 1297 1288 198.131 0 - 0.248248
Total time run: 26.829870
Total writes made: 1297
Write size: 4194304
Bandwidth (MB/sec): 193.367
Average Latency: 0.295922
Max latency: 7.36701
Min latency: 0.033773
Strange. I'm wondering if this has something to do with cache (that is,
operation I could have done before on nodes, as all my nodes are just
freshly rebooted).
Cheers,
--
Yann Dupont - Service IRTS, DSI Université de Nantes
Tel : 02.53.48.49.20 - Mail/Jabber : Yann.Dupont@univ-nantes.fr
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2012-05-31 13:21 UTC|newest]
Thread overview: 73+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-05-24 14:10 poor OSD performance using kernel 3.4 Stefan Priebe - Profihost AG
2012-05-24 14:57 ` Mark Nelson
[not found] ` <CAJCPpW+SKnnVUaDEAsCkKyZwMVrHCRJF2C8zqB4eORgwW5p=1Q@mail.gmail.com>
[not found] ` <4FBE7ABC.5020502@profihost.ag>
2012-05-24 18:53 ` Mark Nelson
2012-05-24 19:05 ` Stefan Priebe
2012-05-25 1:53 ` Mark Nelson
2012-05-25 8:19 ` Stefan Priebe - Profihost AG
2012-05-25 11:31 ` Stefan Priebe - Profihost AG
2012-05-25 12:10 ` Stefan Priebe - Profihost AG
2012-05-25 15:47 ` Alexandre DERUMIER
2012-05-27 9:11 ` Stefan Priebe - Profihost AG
2012-05-27 11:33 ` Alexandre DERUMIER
2012-05-27 18:57 ` Stefan Priebe
2012-05-28 5:37 ` Alexandre DERUMIER
2012-05-28 6:25 ` Stefan Priebe
2012-05-28 6:52 ` Alexandre DERUMIER
2012-05-28 19:48 ` Stefan Priebe
2012-05-29 3:54 ` Alexandre DERUMIER
2012-05-29 8:22 ` Stefan Priebe - Profihost AG
2012-05-29 13:01 ` Alexandre DERUMIER
2012-05-29 14:18 ` Stefan Priebe - Profihost AG
2012-05-29 9:46 ` Stefan Priebe - Profihost AG
2012-05-29 13:39 ` Yann Dupont
2012-05-29 14:43 ` Stefan Priebe - Profihost AG
2012-05-29 17:50 ` Mark Nelson
2012-05-29 19:50 ` Yann Dupont
2012-05-29 21:04 ` Stefan Priebe
2012-05-29 21:08 ` Stefan Priebe
2012-05-29 21:31 ` Yann Dupont
2012-05-29 21:34 ` Stefan Priebe
2012-05-29 21:45 ` Yann Dupont
2012-05-30 6:29 ` Stefan Priebe - Profihost AG
2012-05-29 21:41 ` Mark Nelson
2012-05-30 6:22 ` Stefan Priebe - Profihost AG
2012-05-30 7:20 ` building test cluster : missing /etc/ceph/client.admin.keyring, need help Alexandre DERUMIER
2012-05-30 7:25 ` Stefan Priebe - Profihost AG
2012-05-30 7:33 ` Alexandre DERUMIER
2012-05-30 7:47 ` Alexandre DERUMIER
2012-05-29 22:25 ` poor OSD performance using kernel 3.4 Mark Nelson
2012-05-30 6:33 ` Stefan Priebe - Profihost AG
[not found] ` <CADdPHGs9dpSh9Oyu+5yDhyYU=Et_-zF5MuYybBuuAN5DgR433A@mail.gmail.com>
2012-05-30 7:16 ` Stefan Priebe - Profihost AG
[not found] ` <CADdPHGuiJqZUCK-0qR_CrOo6GRhkjaCdkOhJ2boq3zD0_voTsA@mail.gmail.com>
2012-05-30 11:04 ` Stefan Priebe - Profihost AG
[not found] ` <CADdPHGuLAL5+hkzq0tigqu355DvPxkhE5sxBhOVZPj=EzDSVtA@mail.gmail.com>
2012-05-30 11:25 ` Stefan Priebe - Profihost AG
2012-05-30 12:17 ` Mark Nelson
2012-05-30 12:41 ` Stefan Priebe - Profihost AG
[not found] ` <CADdPHGsmr8Ht1pTWH1Oe8=NmAyM81SSdH+c_GV89D8ntfyUmgA@mail.gmail.com>
2012-05-30 13:19 ` Stefan Priebe - Profihost AG
[not found] ` <CADdPHGvxCmuViy+0==Vkdz_QjC1K+kD5kD1m7+0tYM2YDTtJbw@mail.gmail.com>
2012-05-30 13:54 ` Stefan Priebe - Profihost AG
[not found] ` <4FC63381.6090300@inktank.com>
2012-05-30 14:53 ` Stefan Priebe
2012-05-30 14:56 ` Mark Nelson
2012-05-30 18:26 ` Stefan Priebe
2012-05-30 19:41 ` Mark Nelson
2012-05-30 13:27 ` Mark Nelson
2012-05-30 13:51 ` Stefan Priebe - Profihost AG
2012-05-30 14:16 ` Mark Nelson
2012-05-30 18:42 ` Stefan Priebe
[not found] ` <CADdPHGuxa7TAyqXcXehb9WgKgkHwkybYTrj2oue_PKsiF+oR3A@mail.gmail.com>
2012-05-30 21:10 ` Stefan Priebe
[not found] ` <CADdPHGutEwoDc=Kcrqcx2ZMO=dqhuoT5iLoP-WxqD+e5ZUmBRA@mail.gmail.com>
2012-05-31 7:10 ` poor OSD performance using kernel 3.4 => problem found Stefan Priebe - Profihost AG
2012-05-31 7:30 ` Yehuda Sadeh
[not found] ` <CADdPHGtz9Jq624DMO6Dve2AcJ9vrnFHbyqRa+qheA+0-y4k++g@mail.gmail.com>
2012-05-31 12:31 ` Mark Nelson
2012-05-31 12:33 ` Stefan Priebe - Profihost AG
2012-05-31 13:21 ` Yann Dupont [this message]
2012-05-31 13:37 ` Stefan Priebe - Profihost AG
2012-05-31 13:45 ` Yann Dupont
2012-05-31 14:42 ` Yann Dupont
2012-05-31 15:32 ` Mark Nelson
2012-05-31 15:43 ` Yann Dupont
2012-05-31 16:14 ` Mark Nelson
2012-05-31 16:29 ` Sage Weil
2012-05-31 16:37 ` Yann Dupont
[not found] ` <CADdPHGv0YjxDQFnZML-55jDj7XxHxaxUZ_FeQ=ReKK6Rs7NNhw@mail.gmail.com>
2012-05-31 8:04 ` Stefan Priebe - Profihost AG
2012-05-31 8:09 ` Stefan Majer
2012-05-31 11:34 ` Stefan Priebe - Profihost AG
2012-05-31 12:18 ` Stefan Priebe - Profihost AG
2012-05-30 11:51 ` poor OSD performance using kernel 3.4 Mark Nelson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4FC77045.6050907@univ-nantes.fr \
--to=yann.dupont@univ-nantes.fr \
--cc=ceph-devel@vger.kernel.org \
--cc=mark.nelson@inktank.com \
--cc=s.priebe@profihost.ag \
--cc=stefan.majer@gmail.com \
--cc=yehuda@inktank.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.