From: Yann Dupont <Yann.Dupont@univ-nantes.fr>
To: Stefan Priebe - Profihost AG <s.priebe@profihost.ag>
Cc: ceph-devel@vger.kernel.org
Subject: Re: poor OSD performance using kernel 3.4
Date: Tue, 29 May 2012 15:39:52 +0200 [thread overview]
Message-ID: <4FC4D1A8.1080001@univ-nantes.fr> (raw)
In-Reply-To: <4FC49B12.8020004@profihost.ag>
On 29/05/2012 11:46, Stefan Priebe - Profihost AG wrote:
> It would be really nice if somebody from inktank can comment this whole
> sitation.
>
Hello.
I think I have the same bug :
My setup is with 8 OSD nodes, 3 MDS (1 active) & 3 MON.
All my machines are debian, using a custom 3.4.0 kernel. Ceph is
0.47.2-1~bpo60+1 (debian package)
root@label5:~# rados -p data bench 20 write -t 16
Maintaining 16 concurrent writes of 4194304 bytes for at least 20 seconds.
sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
0 0 0 0 0 0 - 0
1 16 99 83 331.9 332 0.059756 0.0946512
2 16 141 125 249.946 168 0.049822 0.212338
3 16 166 150 199.963 100 0.057352 0.257179
4 16 227 211 210.965 244 0.043592 0.265005
5 16 257 241 192.767 120 0.040883 0.276718
6 16 260 244 162.641 12 1.59593 0.293439
7 16 319 303 173.118 236 0.056913 0.357856
8 16 348 332 165.976 116 0.052954 0.332424
9 16 348 332 147.535 0 - 0.332424
10 16 472 456 182.374 248 0.038543 0.343745
11 16 485 469 170.522 52 0.040475 0.347328
12 16 485 469 156.312 0 - 0.347328
13 16 517 501 154.133 64 0.047759 0.378595
14 16 562 546 155.98 180 0.042814 0.395036
15 16 563 547 145.847 4 0.045834 0.394398
16 16 563 547 136.732 0 - 0.394398
17 16 563 547 128.689 0 - 0.394398
18 16 667 651 144.648 138.667 0.06501 0.440847
19 16 703 687 144.613 144 0.040772 0.421935
min lat: 0.030505 max lat: 5.05834 avg lat: 0.421935
sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
20 16 703 687 137.382 0 - 0.421935
21 16 704 688 131.031 2 2.65675 0.425184
22 14 704 690 125.439 8 3.26857 0.433417
Total time run: 22.042041
Total writes made: 704
Write size: 4194304
Bandwidth (MB/sec): 127.756
Average Latency: 0.498932
Max latency: 5.05834
Min latency: 0.030505
What puzzle me is if I test with pool rbd instead :
root@label5:~# rados -p rbd bench 20 write -t 16
Maintaining 16 concurrent writes of 4194304 bytes for at least 20 seconds.
sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
0 0 0 0 0 0 - 0
1 16 191 175 699.782 700 0.236737 0.0841979
2 16 397 381 761.837 824 0.065643 0.0813094
3 16 602 586 781.193 820 0.07921 0.0808584
4 16 815 799 798.88 852 0.066597 0.0785906
5 16 1026 1010 807.885 844 0.10364 0.0785475
6 16 1249 1233 821.886 892 0.069324 0.0773951
7 16 1461 1445 825.608 848 0.053176 0.0770628
8 16 1680 1664 831.895 876 0.09612 0.0765263
9 16 1897 1881 835.891 868 0.100736 0.0761617
10 16 2105 2089 835.491 832 0.114913 0.0761897
11 16 2329 2313 840.983 896 0.042009 0.0758589
12 16 2553 2537 845.559 896 0.07017 0.0754364
13 16 2786 2770 852.203 932 0.066365 0.0749136
14 16 3009 2993 855.041 892 0.06491 0.0746046
15 16 3228 3212 856.431 876 0.05698 0.0745573
16 16 3437 3421 855.148 836 0.062162 0.0746339
17 16 3652 3636 855.428 860 0.140451 0.074534
18 16 3878 3862 858.121 904 0.081505 0.0743125
19 16 4106 4090 860.952 912 0.079922 0.0742146
min lat: 0.032342 max lat: 0.63151 avg lat: 0.0741575
sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
20 16 4324 4308 861.495 872 0.06199 0.0741575
Total time run: 20.102264
Total writes made: 4325
Write size: 4194304
Bandwidth (MB/sec): 860.600
Average Latency: 0.0743131
Max latency: 0.63151
Min latency: 0.032342
As you can see, much more stable bandwith with this pool.
I understand data & rbd pool probably don't use the same internals, but
is this difference expected ?
disclaimer: By no mean I'm a ceph expert, I'm just experimenting with
it, and still don't understand all the internals.
Cheers,
--
Yann Dupont - Service IRTS, DSI Université de Nantes
Tel : 02.53.48.49.20 - Mail/Jabber : Yann.Dupont@univ-nantes.fr
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2012-05-29 13:40 UTC|newest]
Thread overview: 73+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-05-24 14:10 poor OSD performance using kernel 3.4 Stefan Priebe - Profihost AG
2012-05-24 14:57 ` Mark Nelson
[not found] ` <CAJCPpW+SKnnVUaDEAsCkKyZwMVrHCRJF2C8zqB4eORgwW5p=1Q@mail.gmail.com>
[not found] ` <4FBE7ABC.5020502@profihost.ag>
2012-05-24 18:53 ` Mark Nelson
2012-05-24 19:05 ` Stefan Priebe
2012-05-25 1:53 ` Mark Nelson
2012-05-25 8:19 ` Stefan Priebe - Profihost AG
2012-05-25 11:31 ` Stefan Priebe - Profihost AG
2012-05-25 12:10 ` Stefan Priebe - Profihost AG
2012-05-25 15:47 ` Alexandre DERUMIER
2012-05-27 9:11 ` Stefan Priebe - Profihost AG
2012-05-27 11:33 ` Alexandre DERUMIER
2012-05-27 18:57 ` Stefan Priebe
2012-05-28 5:37 ` Alexandre DERUMIER
2012-05-28 6:25 ` Stefan Priebe
2012-05-28 6:52 ` Alexandre DERUMIER
2012-05-28 19:48 ` Stefan Priebe
2012-05-29 3:54 ` Alexandre DERUMIER
2012-05-29 8:22 ` Stefan Priebe - Profihost AG
2012-05-29 13:01 ` Alexandre DERUMIER
2012-05-29 14:18 ` Stefan Priebe - Profihost AG
2012-05-29 9:46 ` Stefan Priebe - Profihost AG
2012-05-29 13:39 ` Yann Dupont [this message]
2012-05-29 14:43 ` Stefan Priebe - Profihost AG
2012-05-29 17:50 ` Mark Nelson
2012-05-29 19:50 ` Yann Dupont
2012-05-29 21:04 ` Stefan Priebe
2012-05-29 21:08 ` Stefan Priebe
2012-05-29 21:31 ` Yann Dupont
2012-05-29 21:34 ` Stefan Priebe
2012-05-29 21:45 ` Yann Dupont
2012-05-30 6:29 ` Stefan Priebe - Profihost AG
2012-05-29 21:41 ` Mark Nelson
2012-05-30 6:22 ` Stefan Priebe - Profihost AG
2012-05-30 7:20 ` building test cluster : missing /etc/ceph/client.admin.keyring, need help Alexandre DERUMIER
2012-05-30 7:25 ` Stefan Priebe - Profihost AG
2012-05-30 7:33 ` Alexandre DERUMIER
2012-05-30 7:47 ` Alexandre DERUMIER
2012-05-29 22:25 ` poor OSD performance using kernel 3.4 Mark Nelson
2012-05-30 6:33 ` Stefan Priebe - Profihost AG
[not found] ` <CADdPHGs9dpSh9Oyu+5yDhyYU=Et_-zF5MuYybBuuAN5DgR433A@mail.gmail.com>
2012-05-30 7:16 ` Stefan Priebe - Profihost AG
[not found] ` <CADdPHGuiJqZUCK-0qR_CrOo6GRhkjaCdkOhJ2boq3zD0_voTsA@mail.gmail.com>
2012-05-30 11:04 ` Stefan Priebe - Profihost AG
[not found] ` <CADdPHGuLAL5+hkzq0tigqu355DvPxkhE5sxBhOVZPj=EzDSVtA@mail.gmail.com>
2012-05-30 11:25 ` Stefan Priebe - Profihost AG
2012-05-30 12:17 ` Mark Nelson
2012-05-30 12:41 ` Stefan Priebe - Profihost AG
[not found] ` <CADdPHGsmr8Ht1pTWH1Oe8=NmAyM81SSdH+c_GV89D8ntfyUmgA@mail.gmail.com>
2012-05-30 13:19 ` Stefan Priebe - Profihost AG
[not found] ` <CADdPHGvxCmuViy+0==Vkdz_QjC1K+kD5kD1m7+0tYM2YDTtJbw@mail.gmail.com>
2012-05-30 13:54 ` Stefan Priebe - Profihost AG
[not found] ` <4FC63381.6090300@inktank.com>
2012-05-30 14:53 ` Stefan Priebe
2012-05-30 14:56 ` Mark Nelson
2012-05-30 18:26 ` Stefan Priebe
2012-05-30 19:41 ` Mark Nelson
2012-05-30 13:27 ` Mark Nelson
2012-05-30 13:51 ` Stefan Priebe - Profihost AG
2012-05-30 14:16 ` Mark Nelson
2012-05-30 18:42 ` Stefan Priebe
[not found] ` <CADdPHGuxa7TAyqXcXehb9WgKgkHwkybYTrj2oue_PKsiF+oR3A@mail.gmail.com>
2012-05-30 21:10 ` Stefan Priebe
[not found] ` <CADdPHGutEwoDc=Kcrqcx2ZMO=dqhuoT5iLoP-WxqD+e5ZUmBRA@mail.gmail.com>
2012-05-31 7:10 ` poor OSD performance using kernel 3.4 => problem found Stefan Priebe - Profihost AG
2012-05-31 7:30 ` Yehuda Sadeh
[not found] ` <CADdPHGtz9Jq624DMO6Dve2AcJ9vrnFHbyqRa+qheA+0-y4k++g@mail.gmail.com>
2012-05-31 12:31 ` Mark Nelson
2012-05-31 12:33 ` Stefan Priebe - Profihost AG
2012-05-31 13:21 ` Yann Dupont
2012-05-31 13:37 ` Stefan Priebe - Profihost AG
2012-05-31 13:45 ` Yann Dupont
2012-05-31 14:42 ` Yann Dupont
2012-05-31 15:32 ` Mark Nelson
2012-05-31 15:43 ` Yann Dupont
2012-05-31 16:14 ` Mark Nelson
2012-05-31 16:29 ` Sage Weil
2012-05-31 16:37 ` Yann Dupont
[not found] ` <CADdPHGv0YjxDQFnZML-55jDj7XxHxaxUZ_FeQ=ReKK6Rs7NNhw@mail.gmail.com>
2012-05-31 8:04 ` Stefan Priebe - Profihost AG
2012-05-31 8:09 ` Stefan Majer
2012-05-31 11:34 ` Stefan Priebe - Profihost AG
2012-05-31 12:18 ` Stefan Priebe - Profihost AG
2012-05-30 11:51 ` poor OSD performance using kernel 3.4 Mark Nelson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4FC4D1A8.1080001@univ-nantes.fr \
--to=yann.dupont@univ-nantes.fr \
--cc=ceph-devel@vger.kernel.org \
--cc=s.priebe@profihost.ag \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox