* 10/7/2014 Weekly Ceph Performance Meeting
@ 2014-10-08 0:51 Mark Nelson
2014-10-08 16:32 ` 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params Andreas Bluemle
0 siblings, 1 reply; 22+ messages in thread
From: Mark Nelson @ 2014-10-08 0:51 UTC (permalink / raw)
To: ceph-devel@vger.kernel.org
Hi All,
Just a remind that the weekly performance meeting is on Wednesdays at
8AM PST. Same bat time, same bat channel!
Etherpad URL:
http://pad.ceph.com/p/performance_weekly
To join the Meeting:
https://bluejeans.com/268261044
To join via Browser:
https://bluejeans.com/268261044/browser
To join with Lync:
https://bluejeans.com/268261044/lync
To join via Room System:
Video Conferencing System: bjn.vc -or- 199.48.152.152
Meeting ID: 268261044
To join via Phone:
1) Dial:
+1 408 740 7256
+1 888 240 2560(US Toll Free)
+1 408 317 9253(Alternate Number)
(see all numbers - http://bluejeans.com/numbers)
2) Enter Conference ID: 268261044
Mark
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
2014-10-08 0:51 10/7/2014 Weekly Ceph Performance Meeting Mark Nelson
@ 2014-10-08 16:32 ` Andreas Bluemle
2014-10-08 17:38 ` Somnath Roy
0 siblings, 1 reply; 22+ messages in thread
From: Andreas Bluemle @ 2014-10-08 16:32 UTC (permalink / raw)
To: ceph-devel@vger.kernel.org
Hi,
as mentioned during today's meeting, here are the kernel
boot parameters which I found to provide the basis for
good performance results:
processor.max_cstate=0
intel_idle.max_cstate=0
I understand these to basically turn off any power saving modes
of the CPU; the CPU's we are using are like
Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
At the BIOS level, we
- turn off Hyperthraeding
- turn off Turbo mode (in order ot not leave the specifications)
- turn on frequency floor override
We also assert that
/sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
is set to "performance"
Using above we see a constant frequency at the maximum level
allowed by the CPU (except Turbo mode).
Best Regards
Andreas Bluemle
On Wed, 8 Oct 2014 02:51:21 +0200
Mark Nelson <mark.nelson@inktank.com> wrote:
> Hi All,
>
> Just a remind that the weekly performance meeting is on Wednesdays at
> 8AM PST. Same bat time, same bat channel!
>
> Etherpad URL:
> http://pad.ceph.com/p/performance_weekly
>
> To join the Meeting:
> https://bluejeans.com/268261044
>
> To join via Browser:
> https://bluejeans.com/268261044/browser
>
> To join with Lync:
> https://bluejeans.com/268261044/lync
>
>
> To join via Room System:
> Video Conferencing System: bjn.vc -or- 199.48.152.152
> Meeting ID: 268261044
>
> To join via Phone:
> 1) Dial:
> +1 408 740 7256
> +1 888 240 2560(US Toll Free)
> +1 408 317 9253(Alternate Number)
> (see all numbers - http://bluejeans.com/numbers)
> 2) Enter Conference ID: 268261044
>
> Mark
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
--
Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de
ITXperts GmbH http://www.itxperts.de
Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917
D-81541 Muenchen (Germany) Fax: (+49) 89 89044910
Company details: http://www.itxperts.de/imprint.htm
^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
2014-10-08 16:32 ` 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params Andreas Bluemle
@ 2014-10-08 17:38 ` Somnath Roy
2014-10-08 17:47 ` Duan, Jiangang
` (3 more replies)
0 siblings, 4 replies; 22+ messages in thread
From: Somnath Roy @ 2014-10-08 17:38 UTC (permalink / raw)
To: Andreas Bluemle, ceph-devel@vger.kernel.org
Thanks Andres for sharing this. I will try those out.
BTW, I am using Ubuntu 14.04 LTS and couldn't find any sysfs entry like 'cpufreq'..
root@stormeap-4:~# ll /sys/devices/system/cpu/cpu10/
cache/ crash_notes driver/ microcode/ online subsystem/ topology/
cpuidle/ crash_notes_size firmware_node/ node0/ power/ thermal_throttle/ uevent
I am using Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz.
Regards
Somnath
-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Andreas Bluemle
Sent: Wednesday, October 08, 2014 9:33 AM
To: ceph-devel@vger.kernel.org
Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Hi,
as mentioned during today's meeting, here are the kernel boot parameters which I found to provide the basis for good performance results:
processor.max_cstate=0
intel_idle.max_cstate=0
I understand these to basically turn off any power saving modes of the CPU; the CPU's we are using are like
Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
At the BIOS level, we
- turn off Hyperthraeding
- turn off Turbo mode (in order ot not leave the specifications)
- turn on frequency floor override
We also assert that
/sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
is set to "performance"
Using above we see a constant frequency at the maximum level allowed by the CPU (except Turbo mode).
Best Regards
Andreas Bluemle
On Wed, 8 Oct 2014 02:51:21 +0200
Mark Nelson <mark.nelson@inktank.com> wrote:
> Hi All,
>
> Just a remind that the weekly performance meeting is on Wednesdays at
> 8AM PST. Same bat time, same bat channel!
>
> Etherpad URL:
> http://pad.ceph.com/p/performance_weekly
>
> To join the Meeting:
> https://bluejeans.com/268261044
>
> To join via Browser:
> https://bluejeans.com/268261044/browser
>
> To join with Lync:
> https://bluejeans.com/268261044/lync
>
>
> To join via Room System:
> Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID:
> 268261044
>
> To join via Phone:
> 1) Dial:
> +1 408 740 7256
> +1 888 240 2560(US Toll Free)
> +1 408 317 9253(Alternate Number)
> (see all numbers - http://bluejeans.com/numbers)
> 2) Enter Conference ID: 268261044
>
> Mark
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org More majordomo
> info at http://vger.kernel.org/majordomo-info.html
>
>
--
Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de
ITXperts GmbH http://www.itxperts.de
Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917
D-81541 Muenchen (Germany) Fax: (+49) 89 89044910
Company details: http://www.itxperts.de/imprint.htm
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
2014-10-08 17:38 ` Somnath Roy
@ 2014-10-08 17:47 ` Duan, Jiangang
2014-10-08 17:53 ` Somnath Roy
2014-10-08 17:57 ` Loic Dachary
` (2 subsequent siblings)
3 siblings, 1 reply; 22+ messages in thread
From: Duan, Jiangang @ 2014-10-08 17:47 UTC (permalink / raw)
To: Somnath Roy, Andreas Bluemle, ceph-devel@vger.kernel.org
Can you guys share the w/ HT and w/o HT data? I want to take a look at that to understand why.
-jiangang
-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Somnath Roy
Sent: Wednesday, October 08, 2014 10:38 AM
To: Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Thanks Andres for sharing this. I will try those out.
BTW, I am using Ubuntu 14.04 LTS and couldn't find any sysfs entry like 'cpufreq'..
root@stormeap-4:~# ll /sys/devices/system/cpu/cpu10/
cache/ crash_notes driver/ microcode/ online subsystem/ topology/
cpuidle/ crash_notes_size firmware_node/ node0/ power/ thermal_throttle/ uevent
I am using Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz.
Regards
Somnath
-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Andreas Bluemle
Sent: Wednesday, October 08, 2014 9:33 AM
To: ceph-devel@vger.kernel.org
Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Hi,
as mentioned during today's meeting, here are the kernel boot parameters which I found to provide the basis for good performance results:
processor.max_cstate=0
intel_idle.max_cstate=0
I understand these to basically turn off any power saving modes of the CPU; the CPU's we are using are like
Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
At the BIOS level, we
- turn off Hyperthraeding
- turn off Turbo mode (in order ot not leave the specifications)
- turn on frequency floor override
We also assert that
/sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
is set to "performance"
Using above we see a constant frequency at the maximum level allowed by the CPU (except Turbo mode).
Best Regards
Andreas Bluemle
On Wed, 8 Oct 2014 02:51:21 +0200
Mark Nelson <mark.nelson@inktank.com> wrote:
> Hi All,
>
> Just a remind that the weekly performance meeting is on Wednesdays at
> 8AM PST. Same bat time, same bat channel!
>
> Etherpad URL:
> http://pad.ceph.com/p/performance_weekly
>
> To join the Meeting:
> https://bluejeans.com/268261044
>
> To join via Browser:
> https://bluejeans.com/268261044/browser
>
> To join with Lync:
> https://bluejeans.com/268261044/lync
>
>
> To join via Room System:
> Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID:
> 268261044
>
> To join via Phone:
> 1) Dial:
> +1 408 740 7256
> +1 888 240 2560(US Toll Free)
> +1 408 317 9253(Alternate Number)
> (see all numbers - http://bluejeans.com/numbers)
> 2) Enter Conference ID: 268261044
>
> Mark
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org More majordomo
> info at http://vger.kernel.org/majordomo-info.html
>
>
--
Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de
ITXperts GmbH http://www.itxperts.de
Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917
D-81541 Muenchen (Germany) Fax: (+49) 89 89044910
Company details: http://www.itxperts.de/imprint.htm
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
2014-10-08 17:47 ` Duan, Jiangang
@ 2014-10-08 17:53 ` Somnath Roy
2014-10-08 20:03 ` Duan, Jiangang
0 siblings, 1 reply; 22+ messages in thread
From: Somnath Roy @ 2014-10-08 17:53 UTC (permalink / raw)
To: Duan, Jiangang, Andreas Bluemle, ceph-devel@vger.kernel.org
Hi Jiangang,
Give me a day or two, I will gather all the data and share with community.
Thanks & Regards
Somnath
-----Original Message-----
From: Duan, Jiangang [mailto:jiangang.duan@intel.com]
Sent: Wednesday, October 08, 2014 10:47 AM
To: Somnath Roy; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Can you guys share the w/ HT and w/o HT data? I want to take a look at that to understand why.
-jiangang
-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Somnath Roy
Sent: Wednesday, October 08, 2014 10:38 AM
To: Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Thanks Andres for sharing this. I will try those out.
BTW, I am using Ubuntu 14.04 LTS and couldn't find any sysfs entry like 'cpufreq'..
root@stormeap-4:~# ll /sys/devices/system/cpu/cpu10/
cache/ crash_notes driver/ microcode/ online subsystem/ topology/
cpuidle/ crash_notes_size firmware_node/ node0/ power/ thermal_throttle/ uevent
I am using Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz.
Regards
Somnath
-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Andreas Bluemle
Sent: Wednesday, October 08, 2014 9:33 AM
To: ceph-devel@vger.kernel.org
Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Hi,
as mentioned during today's meeting, here are the kernel boot parameters which I found to provide the basis for good performance results:
processor.max_cstate=0
intel_idle.max_cstate=0
I understand these to basically turn off any power saving modes of the CPU; the CPU's we are using are like
Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
At the BIOS level, we
- turn off Hyperthraeding
- turn off Turbo mode (in order ot not leave the specifications)
- turn on frequency floor override
We also assert that
/sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
is set to "performance"
Using above we see a constant frequency at the maximum level allowed by the CPU (except Turbo mode).
Best Regards
Andreas Bluemle
On Wed, 8 Oct 2014 02:51:21 +0200
Mark Nelson <mark.nelson@inktank.com> wrote:
> Hi All,
>
> Just a remind that the weekly performance meeting is on Wednesdays at
> 8AM PST. Same bat time, same bat channel!
>
> Etherpad URL:
> http://pad.ceph.com/p/performance_weekly
>
> To join the Meeting:
> https://bluejeans.com/268261044
>
> To join via Browser:
> https://bluejeans.com/268261044/browser
>
> To join with Lync:
> https://bluejeans.com/268261044/lync
>
>
> To join via Room System:
> Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID:
> 268261044
>
> To join via Phone:
> 1) Dial:
> +1 408 740 7256
> +1 888 240 2560(US Toll Free)
> +1 408 317 9253(Alternate Number)
> (see all numbers - http://bluejeans.com/numbers)
> 2) Enter Conference ID: 268261044
>
> Mark
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org More majordomo
> info at http://vger.kernel.org/majordomo-info.html
>
>
--
Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de
ITXperts GmbH http://www.itxperts.de
Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917
D-81541 Muenchen (Germany) Fax: (+49) 89 89044910
Company details: http://www.itxperts.de/imprint.htm
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
2014-10-08 17:38 ` Somnath Roy
2014-10-08 17:47 ` Duan, Jiangang
@ 2014-10-08 17:57 ` Loic Dachary
2014-10-08 18:07 ` Alexandre DERUMIER
2014-10-08 18:35 ` Stefan Priebe
3 siblings, 0 replies; 22+ messages in thread
From: Loic Dachary @ 2014-10-08 17:57 UTC (permalink / raw)
To: Somnath Roy, Andreas Bluemle, ceph-devel@vger.kernel.org
[-- Attachment #1: Type: text/plain, Size: 5287 bytes --]
Hi,
On 08/10/2014 19:38, Somnath Roy wrote:
> Thanks Andres for sharing this. I will try those out.
> BTW, I am using Ubuntu 14.04 LTS and couldn't find any sysfs entry like 'cpufreq'..
>
> root@stormeap-4:~# ll /sys/devices/system/cpu/cpu10/
> cache/ crash_notes driver/ microcode/ online subsystem/ topology/
> cpuidle/ crash_notes_size firmware_node/ node0/ power/ thermal_throttle/ uevent
>
> I am using Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz.
-rw-r--r-- 1 root root 4096 oct. 8 17:31 /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
-rw-r--r-- 1 root root 4096 oct. 8 17:31 /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor
-rw-r--r-- 1 root root 4096 oct. 8 17:31 /sys/devices/system/cpu/cpu2/cpufreq/scaling_governor
-rw-r--r-- 1 root root 4096 oct. 8 17:31 /sys/devices/system/cpu/cpu3/cpufreq/scaling_governor
-rw-r--r-- 1 root root 4096 oct. 8 17:31 /sys/devices/system/cpu/cpu4/cpufreq/scaling_governor
-rw-r--r-- 1 root root 4096 oct. 8 17:31 /sys/devices/system/cpu/cpu5/cpufreq/scaling_governor
-rw-r--r-- 1 root root 4096 oct. 8 17:31 /sys/devices/system/cpu/cpu6/cpufreq/scaling_governor
-rw-r--r-- 1 root root 4096 oct. 8 17:31 /sys/devices/system/cpu/cpu7/cpufreq/scaling_governor
model name : Intel(R) Core(TM) i7-4900MQ CPU @ 2.80GHz
$ lsb_release -d
Description: Ubuntu Trusty Tahr (development branch)
Cheers
>
> Regards
> Somnath
>
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Andreas Bluemle
> Sent: Wednesday, October 08, 2014 9:33 AM
> To: ceph-devel@vger.kernel.org
> Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
>
> Hi,
>
> as mentioned during today's meeting, here are the kernel boot parameters which I found to provide the basis for good performance results:
>
> processor.max_cstate=0
> intel_idle.max_cstate=0
>
> I understand these to basically turn off any power saving modes of the CPU; the CPU's we are using are like
> Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
> Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
>
> At the BIOS level, we
> - turn off Hyperthraeding
> - turn off Turbo mode (in order ot not leave the specifications)
> - turn on frequency floor override
>
> We also assert that
> /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
> is set to "performance"
>
> Using above we see a constant frequency at the maximum level allowed by the CPU (except Turbo mode).
>
>
> Best Regards
>
> Andreas Bluemle
>
>
>
>
>
>
> On Wed, 8 Oct 2014 02:51:21 +0200
> Mark Nelson <mark.nelson@inktank.com> wrote:
>
>> Hi All,
>>
>> Just a remind that the weekly performance meeting is on Wednesdays at
>> 8AM PST. Same bat time, same bat channel!
>>
>> Etherpad URL:
>> http://pad.ceph.com/p/performance_weekly
>>
>> To join the Meeting:
>> https://bluejeans.com/268261044
>>
>> To join via Browser:
>> https://bluejeans.com/268261044/browser
>>
>> To join with Lync:
>> https://bluejeans.com/268261044/lync
>>
>>
>> To join via Room System:
>> Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID:
>> 268261044
>>
>> To join via Phone:
>> 1) Dial:
>> +1 408 740 7256
>> +1 888 240 2560(US Toll Free)
>> +1 408 317 9253(Alternate Number)
>> (see all numbers - http://bluejeans.com/numbers)
>> 2) Enter Conference ID: 268261044
>>
>> Mark
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>> in the body of a message to majordomo@vger.kernel.org More majordomo
>> info at http://vger.kernel.org/majordomo-info.html
>>
>>
>
>
>
> --
> Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de
> ITXperts GmbH http://www.itxperts.de
> Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917
> D-81541 Muenchen (Germany) Fax: (+49) 89 89044910
>
> Company details: http://www.itxperts.de/imprint.htm
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> ________________________________
>
> PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
Loïc Dachary, Artisan Logiciel Libre
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
2014-10-08 17:38 ` Somnath Roy
2014-10-08 17:47 ` Duan, Jiangang
2014-10-08 17:57 ` Loic Dachary
@ 2014-10-08 18:07 ` Alexandre DERUMIER
2014-10-08 18:35 ` Stefan Priebe
3 siblings, 0 replies; 22+ messages in thread
From: Alexandre DERUMIER @ 2014-10-08 18:07 UTC (permalink / raw)
To: Somnath Roy; +Cc: Andreas Bluemle, ceph-devel
hi,
>>BTW, I am using Ubuntu 14.04 LTS and couldn't find any sysfs entry like 'cpufreq'..
check this arch wiki about kernel modules needed
https://wiki.archlinux.org/index.php/CPU_frequency_scaling
Also note that all theses tuning can normaly be done at bios level.
(on last dell servers bios, setting power profile to max performance, is setting the governor to max and disable all cstate)
I'm always doing it on my kvm hypervisors hosts.
They are also the C1E option to disable on AMD processor.
----- Mail original -----
De: "Somnath Roy" <Somnath.Roy@sandisk.com>
À: "Andreas Bluemle" <andreas.bluemle@itxperts.de>, ceph-devel@vger.kernel.org
Envoyé: Mercredi 8 Octobre 2014 19:38:26
Objet: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Thanks Andres for sharing this. I will try those out.
BTW, I am using Ubuntu 14.04 LTS and couldn't find any sysfs entry like 'cpufreq'..
root@stormeap-4:~# ll /sys/devices/system/cpu/cpu10/
cache/ crash_notes driver/ microcode/ online subsystem/ topology/
cpuidle/ crash_notes_size firmware_node/ node0/ power/ thermal_throttle/ uevent
I am using Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz.
Regards
Somnath
-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Andreas Bluemle
Sent: Wednesday, October 08, 2014 9:33 AM
To: ceph-devel@vger.kernel.org
Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Hi,
as mentioned during today's meeting, here are the kernel boot parameters which I found to provide the basis for good performance results:
processor.max_cstate=0
intel_idle.max_cstate=0
I understand these to basically turn off any power saving modes of the CPU; the CPU's we are using are like
Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
At the BIOS level, we
- turn off Hyperthraeding
- turn off Turbo mode (in order ot not leave the specifications)
- turn on frequency floor override
We also assert that
/sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
is set to "performance"
Using above we see a constant frequency at the maximum level allowed by the CPU (except Turbo mode).
Best Regards
Andreas Bluemle
On Wed, 8 Oct 2014 02:51:21 +0200
Mark Nelson <mark.nelson@inktank.com> wrote:
> Hi All,
>
> Just a remind that the weekly performance meeting is on Wednesdays at
> 8AM PST. Same bat time, same bat channel!
>
> Etherpad URL:
> http://pad.ceph.com/p/performance_weekly
>
> To join the Meeting:
> https://bluejeans.com/268261044
>
> To join via Browser:
> https://bluejeans.com/268261044/browser
>
> To join with Lync:
> https://bluejeans.com/268261044/lync
>
>
> To join via Room System:
> Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID:
> 268261044
>
> To join via Phone:
> 1) Dial:
> +1 408 740 7256
> +1 888 240 2560(US Toll Free)
> +1 408 317 9253(Alternate Number)
> (see all numbers - http://bluejeans.com/numbers)
> 2) Enter Conference ID: 268261044
>
> Mark
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org More majordomo
> info at http://vger.kernel.org/majordomo-info.html
>
>
--
Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de
ITXperts GmbH http://www.itxperts.de
Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917
D-81541 Muenchen (Germany) Fax: (+49) 89 89044910
Company details: http://www.itxperts.de/imprint.htm
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
2014-10-08 17:38 ` Somnath Roy
` (2 preceding siblings ...)
2014-10-08 18:07 ` Alexandre DERUMIER
@ 2014-10-08 18:35 ` Stefan Priebe
2014-10-08 23:55 ` Paul Von-Stamwitz
3 siblings, 1 reply; 22+ messages in thread
From: Stefan Priebe @ 2014-10-08 18:35 UTC (permalink / raw)
To: Somnath Roy, Andreas Bluemle, ceph-devel@vger.kernel.org
> Hi,
>
> as mentioned during today's meeting, here are the kernel boot parameters which I found to provide the basis for good performance results:
>
> processor.max_cstate=0
> intel_idle.max_cstate=0
>
> I understand these to basically turn off any power saving modes of the CPU; the CPU's we are using are like
> Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
> Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
>
> At the BIOS level, we
> - turn off Hyperthraeding
> - turn off Turbo mode (in order ot not leave the specifications)
> - turn on frequency floor override
>
> We also assert that
> /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
> is set to "performance"
>
> Using above we see a constant frequency at the maximum level allowed by the CPU (except Turbo mode).
How much performance do we gain by this? Till now i thought it's just
1-3% so i'm still running ondemand govenor plus power savings.
Greets,
Stefan
> Best Regards
>
> Andreas Bluemle
>
>
>
>
>
>
> On Wed, 8 Oct 2014 02:51:21 +0200
> Mark Nelson <mark.nelson@inktank.com> wrote:
>
>> Hi All,
>>
>> Just a remind that the weekly performance meeting is on Wednesdays at
>> 8AM PST. Same bat time, same bat channel!
>>
>> Etherpad URL:
>> http://pad.ceph.com/p/performance_weekly
>>
>> To join the Meeting:
>> https://bluejeans.com/268261044
>>
>> To join via Browser:
>> https://bluejeans.com/268261044/browser
>>
>> To join with Lync:
>> https://bluejeans.com/268261044/lync
>>
>>
>> To join via Room System:
>> Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID:
>> 268261044
>>
>> To join via Phone:
>> 1) Dial:
>> +1 408 740 7256
>> +1 888 240 2560(US Toll Free)
>> +1 408 317 9253(Alternate Number)
>> (see all numbers - http://bluejeans.com/numbers)
>> 2) Enter Conference ID: 268261044
>>
>> Mark
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>> in the body of a message to majordomo@vger.kernel.org More majordomo
>> info at http://vger.kernel.org/majordomo-info.html
>>
>>
>
>
>
> --
> Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de
> ITXperts GmbH http://www.itxperts.de
> Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917
> D-81541 Muenchen (Germany) Fax: (+49) 89 89044910
>
> Company details: http://www.itxperts.de/imprint.htm
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> ________________________________
>
> PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
2014-10-08 17:53 ` Somnath Roy
@ 2014-10-08 20:03 ` Duan, Jiangang
2014-10-09 0:50 ` Somnath Roy
0 siblings, 1 reply; 22+ messages in thread
From: Duan, Jiangang @ 2014-10-08 20:03 UTC (permalink / raw)
To: Somnath Roy, Andreas Bluemle, ceph-devel@vger.kernel.org
Sound good. Thanks. -jiangang
-----Original Message-----
From: Somnath Roy [mailto:Somnath.Roy@sandisk.com]
Sent: Wednesday, October 08, 2014 10:53 AM
To: Duan, Jiangang; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Hi Jiangang,
Give me a day or two, I will gather all the data and share with community.
Thanks & Regards
Somnath
-----Original Message-----
From: Duan, Jiangang [mailto:jiangang.duan@intel.com]
Sent: Wednesday, October 08, 2014 10:47 AM
To: Somnath Roy; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Can you guys share the w/ HT and w/o HT data? I want to take a look at that to understand why.
-jiangang
-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Somnath Roy
Sent: Wednesday, October 08, 2014 10:38 AM
To: Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Thanks Andres for sharing this. I will try those out.
BTW, I am using Ubuntu 14.04 LTS and couldn't find any sysfs entry like 'cpufreq'..
root@stormeap-4:~# ll /sys/devices/system/cpu/cpu10/
cache/ crash_notes driver/ microcode/ online subsystem/ topology/
cpuidle/ crash_notes_size firmware_node/ node0/ power/ thermal_throttle/ uevent
I am using Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz.
Regards
Somnath
-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Andreas Bluemle
Sent: Wednesday, October 08, 2014 9:33 AM
To: ceph-devel@vger.kernel.org
Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Hi,
as mentioned during today's meeting, here are the kernel boot parameters which I found to provide the basis for good performance results:
processor.max_cstate=0
intel_idle.max_cstate=0
I understand these to basically turn off any power saving modes of the CPU; the CPU's we are using are like
Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
At the BIOS level, we
- turn off Hyperthraeding
- turn off Turbo mode (in order ot not leave the specifications)
- turn on frequency floor override
We also assert that
/sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
is set to "performance"
Using above we see a constant frequency at the maximum level allowed by the CPU (except Turbo mode).
Best Regards
Andreas Bluemle
On Wed, 8 Oct 2014 02:51:21 +0200
Mark Nelson <mark.nelson@inktank.com> wrote:
> Hi All,
>
> Just a remind that the weekly performance meeting is on Wednesdays at
> 8AM PST. Same bat time, same bat channel!
>
> Etherpad URL:
> http://pad.ceph.com/p/performance_weekly
>
> To join the Meeting:
> https://bluejeans.com/268261044
>
> To join via Browser:
> https://bluejeans.com/268261044/browser
>
> To join with Lync:
> https://bluejeans.com/268261044/lync
>
>
> To join via Room System:
> Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID:
> 268261044
>
> To join via Phone:
> 1) Dial:
> +1 408 740 7256
> +1 888 240 2560(US Toll Free)
> +1 408 317 9253(Alternate Number)
> (see all numbers - http://bluejeans.com/numbers)
> 2) Enter Conference ID: 268261044
>
> Mark
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org More majordomo
> info at http://vger.kernel.org/majordomo-info.html
>
>
--
Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de
ITXperts GmbH http://www.itxperts.de
Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917
D-81541 Muenchen (Germany) Fax: (+49) 89 89044910
Company details: http://www.itxperts.de/imprint.htm
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
2014-10-08 18:35 ` Stefan Priebe
@ 2014-10-08 23:55 ` Paul Von-Stamwitz
2014-10-14 11:22 ` Andreas Bluemle
0 siblings, 1 reply; 22+ messages in thread
From: Paul Von-Stamwitz @ 2014-10-08 23:55 UTC (permalink / raw)
To: Stefan Priebe, Somnath Roy, Andreas Bluemle,
ceph-devel@vger.kernel.org
> > Hi,
> >
> > as mentioned during today's meeting, here are the kernel boot parameters
> which I found to provide the basis for good performance results:
> >
> > processor.max_cstate=0
> > intel_idle.max_cstate=0
> >
> > I understand these to basically turn off any power saving modes of the
> CPU; the CPU's we are using are like
> > Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
> > Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
> >
> > At the BIOS level, we
> > - turn off Hyperthraeding
> > - turn off Turbo mode (in order ot not leave the specifications)
> > - turn on frequency floor override
> >
> > We also assert that
> > /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
> > is set to "performance"
> >
> > Using above we see a constant frequency at the maximum level allowed by
> the CPU (except Turbo mode).
>
> How much performance do we gain by this? Till now i thought it's just 1-3% so
> i'm still running ondemand govenor plus power savings.
As always, it depends. I saw noticeable increases in some throughput tests (though I can't recall the % gain.) More important to me was that it made my fio results much more consistent. As we measure improvements, these settings remove some of the "system noise".
Best,
Paul
>
> Greets,
> Stefan
>
> > Best Regards
> >
> > Andreas Bluemle
> >
> >
> >
> >
> >
> >
> > On Wed, 8 Oct 2014 02:51:21 +0200
> > Mark Nelson <mark.nelson@inktank.com> wrote:
> >
> >> Hi All,
> >>
> >> Just a remind that the weekly performance meeting is on Wednesdays at
> >> 8AM PST. Same bat time, same bat channel!
> >>
> >> Etherpad URL:
> >> http://pad.ceph.com/p/performance_weekly
> >>
> >> To join the Meeting:
> >> https://bluejeans.com/268261044
> >>
> >> To join via Browser:
> >> https://bluejeans.com/268261044/browser
> >>
> >> To join with Lync:
> >> https://bluejeans.com/268261044/lync
> >>
> >>
> >> To join via Room System:
> >> Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID:
> >> 268261044
> >>
> >> To join via Phone:
> >> 1) Dial:
> >> +1 408 740 7256
> >> +1 888 240 2560(US Toll Free)
> >> +1 408 317 9253(Alternate Number)
> >> (see all numbers - http://bluejeans.com/numbers)
> >> 2) Enter Conference ID: 268261044
> >>
> >> Mark
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> >> in the body of a message to majordomo@vger.kernel.org More
> majordomo
> >> info at http://vger.kernel.org/majordomo-info.html
> >>
> >>
> >
> >
> >
> > --
> > Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de
> > ITXperts GmbH http://www.itxperts.de
> > Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917
> > D-81541 Muenchen (Germany) Fax: (+49) 89 89044910
> >
> > Company details: http://www.itxperts.de/imprint.htm
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > in the body of a message to majordomo@vger.kernel.org More
> majordomo
> > info at http://vger.kernel.org/majordomo-info.html
> >
> > ________________________________
> >
> > PLEASE NOTE: The information contained in this electronic mail message is
> intended only for the use of the designated recipient(s) named above. If the
> reader of this message is not the intended recipient, you are hereby notified
> that you have received this message in error and that any review,
> dissemination, distribution, or copying of this message is strictly prohibited. If
> you have received this communication in error, please notify the sender by
> telephone or e-mail (as shown above) immediately and destroy any and all
> copies of this message in your possession (whether hard copies or
> electronically stored copies).
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > in the body of a message to majordomo@vger.kernel.org More
> majordomo
> > info at http://vger.kernel.org/majordomo-info.html
> >
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the
> body of a message to majordomo@vger.kernel.org More majordomo info at
> http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
2014-10-08 20:03 ` Duan, Jiangang
@ 2014-10-09 0:50 ` Somnath Roy
2014-10-09 1:07 ` Mark Nelson
` (2 more replies)
0 siblings, 3 replies; 22+ messages in thread
From: Somnath Roy @ 2014-10-09 0:50 UTC (permalink / raw)
To: Duan, Jiangang, Andreas Bluemle, ceph-devel@vger.kernel.org
Hi Jiangang,
I managed to get some data for you but it's for a 3 node cluster. I will try to get data for single node as well.
Test config:
-------------
Cluster and rbd node config:
----------------------------------
"2x E5-2680 10C 2.8GHz 25M
8x 16GB RDIMM, dual rank x4 (128GB)
Mellanox MT27500 40 Gigabit Ethernet
LSI 9207 SAS HBA"
8 X 800 GB SSDs (Optimus Eco) per cluster node
3 cluster nodes + 3 rbd nodes
Total storage ~ 19 TB
We have total 24 OSDs running , each node has 8 OSDs/SSD
Configured 3 pools with 528 PGs/pool and 6 RBDs/pool . Each RBD image size is ~230G.
We have tried on 64K_RR_QD64 workload here.
HT_ENABLE
--------------
IOPS : 112500
Throughput (MB/S): 7012
Avg Resp.Time (m.sec): 17
Max Resp.Time (m.sec): 3184
HT_DISABLE
--------------
IOPS : 120864
Throughput (MB/S): 7530
Avg Resp.Time (m.sec): 11
Max Resp.Time (m.sec): 1056
So, ~7% iop increase but response time decrease is ~35% which is real good.
Thanks & Regards
Somnath
-----Original Message-----
From: Duan, Jiangang [mailto:jiangang.duan@intel.com]
Sent: Wednesday, October 08, 2014 1:03 PM
To: Somnath Roy; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Sound good. Thanks. -jiangang
-----Original Message-----
From: Somnath Roy [mailto:Somnath.Roy@sandisk.com]
Sent: Wednesday, October 08, 2014 10:53 AM
To: Duan, Jiangang; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Hi Jiangang,
Give me a day or two, I will gather all the data and share with community.
Thanks & Regards
Somnath
-----Original Message-----
From: Duan, Jiangang [mailto:jiangang.duan@intel.com]
Sent: Wednesday, October 08, 2014 10:47 AM
To: Somnath Roy; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Can you guys share the w/ HT and w/o HT data? I want to take a look at that to understand why.
-jiangang
-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Somnath Roy
Sent: Wednesday, October 08, 2014 10:38 AM
To: Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Thanks Andres for sharing this. I will try those out.
BTW, I am using Ubuntu 14.04 LTS and couldn't find any sysfs entry like 'cpufreq'..
root@stormeap-4:~# ll /sys/devices/system/cpu/cpu10/
cache/ crash_notes driver/ microcode/ online subsystem/ topology/
cpuidle/ crash_notes_size firmware_node/ node0/ power/ thermal_throttle/ uevent
I am using Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz.
Regards
Somnath
-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Andreas Bluemle
Sent: Wednesday, October 08, 2014 9:33 AM
To: ceph-devel@vger.kernel.org
Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Hi,
as mentioned during today's meeting, here are the kernel boot parameters which I found to provide the basis for good performance results:
processor.max_cstate=0
intel_idle.max_cstate=0
I understand these to basically turn off any power saving modes of the CPU; the CPU's we are using are like
Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
At the BIOS level, we
- turn off Hyperthraeding
- turn off Turbo mode (in order ot not leave the specifications)
- turn on frequency floor override
We also assert that
/sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
is set to "performance"
Using above we see a constant frequency at the maximum level allowed by the CPU (except Turbo mode).
Best Regards
Andreas Bluemle
On Wed, 8 Oct 2014 02:51:21 +0200
Mark Nelson <mark.nelson@inktank.com> wrote:
> Hi All,
>
> Just a remind that the weekly performance meeting is on Wednesdays at
> 8AM PST. Same bat time, same bat channel!
>
> Etherpad URL:
> http://pad.ceph.com/p/performance_weekly
>
> To join the Meeting:
> https://bluejeans.com/268261044
>
> To join via Browser:
> https://bluejeans.com/268261044/browser
>
> To join with Lync:
> https://bluejeans.com/268261044/lync
>
>
> To join via Room System:
> Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID:
> 268261044
>
> To join via Phone:
> 1) Dial:
> +1 408 740 7256
> +1 888 240 2560(US Toll Free)
> +1 408 317 9253(Alternate Number)
> (see all numbers - http://bluejeans.com/numbers)
> 2) Enter Conference ID: 268261044
>
> Mark
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org More majordomo
> info at http://vger.kernel.org/majordomo-info.html
>
>
--
Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de
ITXperts GmbH http://www.itxperts.de
Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917
D-81541 Muenchen (Germany) Fax: (+49) 89 89044910
Company details: http://www.itxperts.de/imprint.htm
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
2014-10-09 0:50 ` Somnath Roy
@ 2014-10-09 1:07 ` Mark Nelson
2014-10-09 6:45 ` Somnath Roy
2014-10-10 23:39 ` Duan, Jiangang
2014-11-05 14:33 ` Zhang, Jian
2 siblings, 1 reply; 22+ messages in thread
From: Mark Nelson @ 2014-10-09 1:07 UTC (permalink / raw)
To: Somnath Roy, Duan, Jiangang, Andreas Bluemle,
ceph-devel@vger.kernel.org
Hi Somnath,
Was this with HT enabled/disabled on both the cluster and the RBD nodes?
Mark
On 10/08/2014 07:50 PM, Somnath Roy wrote:
> Hi Jiangang,
> I managed to get some data for you but it's for a 3 node cluster. I will try to get data for single node as well.
>
> Test config:
> -------------
>
> Cluster and rbd node config:
> ----------------------------------
> "2x E5-2680 10C 2.8GHz 25M
> 8x 16GB RDIMM, dual rank x4 (128GB)
> Mellanox MT27500 40 Gigabit Ethernet
> LSI 9207 SAS HBA"
>
> 8 X 800 GB SSDs (Optimus Eco) per cluster node
>
> 3 cluster nodes + 3 rbd nodes
>
> Total storage ~ 19 TB
>
> We have total 24 OSDs running , each node has 8 OSDs/SSD
>
> Configured 3 pools with 528 PGs/pool and 6 RBDs/pool . Each RBD image size is ~230G.
>
> We have tried on 64K_RR_QD64 workload here.
>
> HT_ENABLE
> --------------
>
> IOPS : 112500
> Throughput (MB/S): 7012
> Avg Resp.Time (m.sec): 17
> Max Resp.Time (m.sec): 3184
>
> HT_DISABLE
> --------------
>
> IOPS : 120864
> Throughput (MB/S): 7530
> Avg Resp.Time (m.sec): 11
> Max Resp.Time (m.sec): 1056
>
>
> So, ~7% iop increase but response time decrease is ~35% which is real good.
>
> Thanks & Regards
> Somnath
>
> -----Original Message-----
> From: Duan, Jiangang [mailto:jiangang.duan@intel.com]
> Sent: Wednesday, October 08, 2014 1:03 PM
> To: Somnath Roy; Andreas Bluemle; ceph-devel@vger.kernel.org
> Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
>
> Sound good. Thanks. -jiangang
>
> -----Original Message-----
> From: Somnath Roy [mailto:Somnath.Roy@sandisk.com]
> Sent: Wednesday, October 08, 2014 10:53 AM
> To: Duan, Jiangang; Andreas Bluemle; ceph-devel@vger.kernel.org
> Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
>
> Hi Jiangang,
> Give me a day or two, I will gather all the data and share with community.
>
> Thanks & Regards
> Somnath
>
> -----Original Message-----
> From: Duan, Jiangang [mailto:jiangang.duan@intel.com]
> Sent: Wednesday, October 08, 2014 10:47 AM
> To: Somnath Roy; Andreas Bluemle; ceph-devel@vger.kernel.org
> Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
>
> Can you guys share the w/ HT and w/o HT data? I want to take a look at that to understand why.
>
> -jiangang
>
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Somnath Roy
> Sent: Wednesday, October 08, 2014 10:38 AM
> To: Andreas Bluemle; ceph-devel@vger.kernel.org
> Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
>
> Thanks Andres for sharing this. I will try those out.
> BTW, I am using Ubuntu 14.04 LTS and couldn't find any sysfs entry like 'cpufreq'..
>
> root@stormeap-4:~# ll /sys/devices/system/cpu/cpu10/
> cache/ crash_notes driver/ microcode/ online subsystem/ topology/
> cpuidle/ crash_notes_size firmware_node/ node0/ power/ thermal_throttle/ uevent
>
> I am using Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz.
>
> Regards
> Somnath
>
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Andreas Bluemle
> Sent: Wednesday, October 08, 2014 9:33 AM
> To: ceph-devel@vger.kernel.org
> Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
>
> Hi,
>
> as mentioned during today's meeting, here are the kernel boot parameters which I found to provide the basis for good performance results:
>
> processor.max_cstate=0
> intel_idle.max_cstate=0
>
> I understand these to basically turn off any power saving modes of the CPU; the CPU's we are using are like
> Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
> Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
>
> At the BIOS level, we
> - turn off Hyperthraeding
> - turn off Turbo mode (in order ot not leave the specifications)
> - turn on frequency floor override
>
> We also assert that
> /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
> is set to "performance"
>
> Using above we see a constant frequency at the maximum level allowed by the CPU (except Turbo mode).
>
>
> Best Regards
>
> Andreas Bluemle
>
>
>
>
>
>
> On Wed, 8 Oct 2014 02:51:21 +0200
> Mark Nelson <mark.nelson@inktank.com> wrote:
>
>> Hi All,
>>
>> Just a remind that the weekly performance meeting is on Wednesdays at
>> 8AM PST. Same bat time, same bat channel!
>>
>> Etherpad URL:
>> http://pad.ceph.com/p/performance_weekly
>>
>> To join the Meeting:
>> https://bluejeans.com/268261044
>>
>> To join via Browser:
>> https://bluejeans.com/268261044/browser
>>
>> To join with Lync:
>> https://bluejeans.com/268261044/lync
>>
>>
>> To join via Room System:
>> Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID:
>> 268261044
>>
>> To join via Phone:
>> 1) Dial:
>> +1 408 740 7256
>> +1 888 240 2560(US Toll Free)
>> +1 408 317 9253(Alternate Number)
>> (see all numbers - http://bluejeans.com/numbers)
>> 2) Enter Conference ID: 268261044
>>
>> Mark
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>> in the body of a message to majordomo@vger.kernel.org More majordomo
>> info at http://vger.kernel.org/majordomo-info.html
>>
>>
>
>
>
> --
> Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de
> ITXperts GmbH http://www.itxperts.de
> Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917
> D-81541 Muenchen (Germany) Fax: (+49) 89 89044910
>
> Company details: http://www.itxperts.de/imprint.htm
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> ________________________________
>
> PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
2014-10-09 1:07 ` Mark Nelson
@ 2014-10-09 6:45 ` Somnath Roy
0 siblings, 0 replies; 22+ messages in thread
From: Somnath Roy @ 2014-10-09 6:45 UTC (permalink / raw)
To: Mark Nelson, Duan, Jiangang, Andreas Bluemle,
ceph-devel@vger.kernel.org
Yes, Mark...
-----Original Message-----
From: Mark Nelson [mailto:mark.nelson@inktank.com]
Sent: Wednesday, October 08, 2014 6:08 PM
To: Somnath Roy; Duan, Jiangang; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Hi Somnath,
Was this with HT enabled/disabled on both the cluster and the RBD nodes?
Mark
On 10/08/2014 07:50 PM, Somnath Roy wrote:
> Hi Jiangang,
> I managed to get some data for you but it's for a 3 node cluster. I will try to get data for single node as well.
>
> Test config:
> -------------
>
> Cluster and rbd node config:
> ----------------------------------
> "2x E5-2680 10C 2.8GHz 25M
> 8x 16GB RDIMM, dual rank x4 (128GB)
> Mellanox MT27500 40 Gigabit Ethernet
> LSI 9207 SAS HBA"
>
> 8 X 800 GB SSDs (Optimus Eco) per cluster node
>
> 3 cluster nodes + 3 rbd nodes
>
> Total storage ~ 19 TB
>
> We have total 24 OSDs running , each node has 8 OSDs/SSD
>
> Configured 3 pools with 528 PGs/pool and 6 RBDs/pool . Each RBD image size is ~230G.
>
> We have tried on 64K_RR_QD64 workload here.
>
> HT_ENABLE
> --------------
>
> IOPS : 112500
> Throughput (MB/S): 7012
> Avg Resp.Time (m.sec): 17
> Max Resp.Time (m.sec): 3184
>
> HT_DISABLE
> --------------
>
> IOPS : 120864
> Throughput (MB/S): 7530
> Avg Resp.Time (m.sec): 11
> Max Resp.Time (m.sec): 1056
>
>
> So, ~7% iop increase but response time decrease is ~35% which is real good.
>
> Thanks & Regards
> Somnath
>
> -----Original Message-----
> From: Duan, Jiangang [mailto:jiangang.duan@intel.com]
> Sent: Wednesday, October 08, 2014 1:03 PM
> To: Somnath Roy; Andreas Bluemle; ceph-devel@vger.kernel.org
> Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot
> params
>
> Sound good. Thanks. -jiangang
>
> -----Original Message-----
> From: Somnath Roy [mailto:Somnath.Roy@sandisk.com]
> Sent: Wednesday, October 08, 2014 10:53 AM
> To: Duan, Jiangang; Andreas Bluemle; ceph-devel@vger.kernel.org
> Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot
> params
>
> Hi Jiangang,
> Give me a day or two, I will gather all the data and share with community.
>
> Thanks & Regards
> Somnath
>
> -----Original Message-----
> From: Duan, Jiangang [mailto:jiangang.duan@intel.com]
> Sent: Wednesday, October 08, 2014 10:47 AM
> To: Somnath Roy; Andreas Bluemle; ceph-devel@vger.kernel.org
> Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot
> params
>
> Can you guys share the w/ HT and w/o HT data? I want to take a look at that to understand why.
>
> -jiangang
>
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org
> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Somnath Roy
> Sent: Wednesday, October 08, 2014 10:38 AM
> To: Andreas Bluemle; ceph-devel@vger.kernel.org
> Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot
> params
>
> Thanks Andres for sharing this. I will try those out.
> BTW, I am using Ubuntu 14.04 LTS and couldn't find any sysfs entry like 'cpufreq'..
>
> root@stormeap-4:~# ll /sys/devices/system/cpu/cpu10/
> cache/ crash_notes driver/ microcode/ online subsystem/ topology/
> cpuidle/ crash_notes_size firmware_node/ node0/ power/ thermal_throttle/ uevent
>
> I am using Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz.
>
> Regards
> Somnath
>
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org
> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Andreas Bluemle
> Sent: Wednesday, October 08, 2014 9:33 AM
> To: ceph-devel@vger.kernel.org
> Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot
> params
>
> Hi,
>
> as mentioned during today's meeting, here are the kernel boot parameters which I found to provide the basis for good performance results:
>
> processor.max_cstate=0
> intel_idle.max_cstate=0
>
> I understand these to basically turn off any power saving modes of the CPU; the CPU's we are using are like
> Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
> Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
>
> At the BIOS level, we
> - turn off Hyperthraeding
> - turn off Turbo mode (in order ot not leave the specifications)
> - turn on frequency floor override
>
> We also assert that
> /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
> is set to "performance"
>
> Using above we see a constant frequency at the maximum level allowed by the CPU (except Turbo mode).
>
>
> Best Regards
>
> Andreas Bluemle
>
>
>
>
>
>
> On Wed, 8 Oct 2014 02:51:21 +0200
> Mark Nelson <mark.nelson@inktank.com> wrote:
>
>> Hi All,
>>
>> Just a remind that the weekly performance meeting is on Wednesdays at
>> 8AM PST. Same bat time, same bat channel!
>>
>> Etherpad URL:
>> http://pad.ceph.com/p/performance_weekly
>>
>> To join the Meeting:
>> https://bluejeans.com/268261044
>>
>> To join via Browser:
>> https://bluejeans.com/268261044/browser
>>
>> To join with Lync:
>> https://bluejeans.com/268261044/lync
>>
>>
>> To join via Room System:
>> Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID:
>> 268261044
>>
>> To join via Phone:
>> 1) Dial:
>> +1 408 740 7256
>> +1 888 240 2560(US Toll Free)
>> +1 408 317 9253(Alternate Number)
>> (see all numbers - http://bluejeans.com/numbers)
>> 2) Enter Conference ID: 268261044
>>
>> Mark
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>> in the body of a message to majordomo@vger.kernel.org More majordomo
>> info at http://vger.kernel.org/majordomo-info.html
>>
>>
>
>
>
> --
> Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de
> ITXperts GmbH http://www.itxperts.de
> Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917
> D-81541 Muenchen (Germany) Fax: (+49) 89 89044910
>
> Company details: http://www.itxperts.de/imprint.htm
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org More majordomo
> info at http://vger.kernel.org/majordomo-info.html
>
> ________________________________
>
> PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org More majordomo
> info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org More majordomo
> info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
2014-10-09 0:50 ` Somnath Roy
2014-10-09 1:07 ` Mark Nelson
@ 2014-10-10 23:39 ` Duan, Jiangang
2014-10-10 23:43 ` Somnath Roy
2014-11-05 14:33 ` Zhang, Jian
2 siblings, 1 reply; 22+ messages in thread
From: Duan, Jiangang @ 2014-10-10 23:39 UTC (permalink / raw)
To: Somnath Roy, Andreas Bluemle, ceph-devel@vger.kernel.org
Thanks. let's try do this test on our setup.
BTW, what workload you use here?
-jiangang
-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Somnath Roy
Sent: Wednesday, October 08, 2014 5:51 PM
To: Duan, Jiangang; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Hi Jiangang,
I managed to get some data for you but it's for a 3 node cluster. I will try to get data for single node as well.
Test config:
-------------
Cluster and rbd node config:
----------------------------------
"2x E5-2680 10C 2.8GHz 25M
8x 16GB RDIMM, dual rank x4 (128GB)
Mellanox MT27500 40 Gigabit Ethernet
LSI 9207 SAS HBA"
8 X 800 GB SSDs (Optimus Eco) per cluster node
3 cluster nodes + 3 rbd nodes
Total storage ~ 19 TB
We have total 24 OSDs running , each node has 8 OSDs/SSD
Configured 3 pools with 528 PGs/pool and 6 RBDs/pool . Each RBD image size is ~230G.
We have tried on 64K_RR_QD64 workload here.
HT_ENABLE
--------------
IOPS : 112500
Throughput (MB/S): 7012
Avg Resp.Time (m.sec): 17
Max Resp.Time (m.sec): 3184
HT_DISABLE
--------------
IOPS : 120864
Throughput (MB/S): 7530
Avg Resp.Time (m.sec): 11
Max Resp.Time (m.sec): 1056
So, ~7% iop increase but response time decrease is ~35% which is real good.
Thanks & Regards
Somnath
-----Original Message-----
From: Duan, Jiangang [mailto:jiangang.duan@intel.com]
Sent: Wednesday, October 08, 2014 1:03 PM
To: Somnath Roy; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Sound good. Thanks. -jiangang
-----Original Message-----
From: Somnath Roy [mailto:Somnath.Roy@sandisk.com]
Sent: Wednesday, October 08, 2014 10:53 AM
To: Duan, Jiangang; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Hi Jiangang,
Give me a day or two, I will gather all the data and share with community.
Thanks & Regards
Somnath
-----Original Message-----
From: Duan, Jiangang [mailto:jiangang.duan@intel.com]
Sent: Wednesday, October 08, 2014 10:47 AM
To: Somnath Roy; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Can you guys share the w/ HT and w/o HT data? I want to take a look at that to understand why.
-jiangang
-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Somnath Roy
Sent: Wednesday, October 08, 2014 10:38 AM
To: Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Thanks Andres for sharing this. I will try those out.
BTW, I am using Ubuntu 14.04 LTS and couldn't find any sysfs entry like 'cpufreq'..
root@stormeap-4:~# ll /sys/devices/system/cpu/cpu10/
cache/ crash_notes driver/ microcode/ online subsystem/ topology/
cpuidle/ crash_notes_size firmware_node/ node0/ power/ thermal_throttle/ uevent
I am using Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz.
Regards
Somnath
-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Andreas Bluemle
Sent: Wednesday, October 08, 2014 9:33 AM
To: ceph-devel@vger.kernel.org
Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Hi,
as mentioned during today's meeting, here are the kernel boot parameters which I found to provide the basis for good performance results:
processor.max_cstate=0
intel_idle.max_cstate=0
I understand these to basically turn off any power saving modes of the CPU; the CPU's we are using are like
Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
At the BIOS level, we
- turn off Hyperthraeding
- turn off Turbo mode (in order ot not leave the specifications)
- turn on frequency floor override
We also assert that
/sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
is set to "performance"
Using above we see a constant frequency at the maximum level allowed by the CPU (except Turbo mode).
Best Regards
Andreas Bluemle
On Wed, 8 Oct 2014 02:51:21 +0200
Mark Nelson <mark.nelson@inktank.com> wrote:
> Hi All,
>
> Just a remind that the weekly performance meeting is on Wednesdays at
> 8AM PST. Same bat time, same bat channel!
>
> Etherpad URL:
> http://pad.ceph.com/p/performance_weekly
>
> To join the Meeting:
> https://bluejeans.com/268261044
>
> To join via Browser:
> https://bluejeans.com/268261044/browser
>
> To join with Lync:
> https://bluejeans.com/268261044/lync
>
>
> To join via Room System:
> Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID:
> 268261044
>
> To join via Phone:
> 1) Dial:
> +1 408 740 7256
> +1 888 240 2560(US Toll Free)
> +1 408 317 9253(Alternate Number)
> (see all numbers - http://bluejeans.com/numbers)
> 2) Enter Conference ID: 268261044
>
> Mark
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org More majordomo
> info at http://vger.kernel.org/majordomo-info.html
>
>
--
Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de
ITXperts GmbH http://www.itxperts.de
Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917
D-81541 Muenchen (Germany) Fax: (+49) 89 89044910
Company details: http://www.itxperts.de/imprint.htm
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
2014-10-10 23:39 ` Duan, Jiangang
@ 2014-10-10 23:43 ` Somnath Roy
0 siblings, 0 replies; 22+ messages in thread
From: Somnath Roy @ 2014-10-10 23:43 UTC (permalink / raw)
To: Duan, Jiangang, Andreas Bluemle, ceph-devel@vger.kernel.org
As I mentioned, total workload is ~19 TB...Each RBD is ~230 GB and io_size = 64K..
-----Original Message-----
From: Duan, Jiangang [mailto:jiangang.duan@intel.com]
Sent: Friday, October 10, 2014 4:39 PM
To: Somnath Roy; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Thanks. let's try do this test on our setup.
BTW, what workload you use here?
-jiangang
-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Somnath Roy
Sent: Wednesday, October 08, 2014 5:51 PM
To: Duan, Jiangang; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Hi Jiangang,
I managed to get some data for you but it's for a 3 node cluster. I will try to get data for single node as well.
Test config:
-------------
Cluster and rbd node config:
----------------------------------
"2x E5-2680 10C 2.8GHz 25M
8x 16GB RDIMM, dual rank x4 (128GB)
Mellanox MT27500 40 Gigabit Ethernet
LSI 9207 SAS HBA"
8 X 800 GB SSDs (Optimus Eco) per cluster node
3 cluster nodes + 3 rbd nodes
Total storage ~ 19 TB
We have total 24 OSDs running , each node has 8 OSDs/SSD
Configured 3 pools with 528 PGs/pool and 6 RBDs/pool . Each RBD image size is ~230G.
We have tried on 64K_RR_QD64 workload here.
HT_ENABLE
--------------
IOPS : 112500
Throughput (MB/S): 7012
Avg Resp.Time (m.sec): 17
Max Resp.Time (m.sec): 3184
HT_DISABLE
--------------
IOPS : 120864
Throughput (MB/S): 7530
Avg Resp.Time (m.sec): 11
Max Resp.Time (m.sec): 1056
So, ~7% iop increase but response time decrease is ~35% which is real good.
Thanks & Regards
Somnath
-----Original Message-----
From: Duan, Jiangang [mailto:jiangang.duan@intel.com]
Sent: Wednesday, October 08, 2014 1:03 PM
To: Somnath Roy; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Sound good. Thanks. -jiangang
-----Original Message-----
From: Somnath Roy [mailto:Somnath.Roy@sandisk.com]
Sent: Wednesday, October 08, 2014 10:53 AM
To: Duan, Jiangang; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Hi Jiangang,
Give me a day or two, I will gather all the data and share with community.
Thanks & Regards
Somnath
-----Original Message-----
From: Duan, Jiangang [mailto:jiangang.duan@intel.com]
Sent: Wednesday, October 08, 2014 10:47 AM
To: Somnath Roy; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Can you guys share the w/ HT and w/o HT data? I want to take a look at that to understand why.
-jiangang
-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Somnath Roy
Sent: Wednesday, October 08, 2014 10:38 AM
To: Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Thanks Andres for sharing this. I will try those out.
BTW, I am using Ubuntu 14.04 LTS and couldn't find any sysfs entry like 'cpufreq'..
root@stormeap-4:~# ll /sys/devices/system/cpu/cpu10/
cache/ crash_notes driver/ microcode/ online subsystem/ topology/
cpuidle/ crash_notes_size firmware_node/ node0/ power/ thermal_throttle/ uevent
I am using Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz.
Regards
Somnath
-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Andreas Bluemle
Sent: Wednesday, October 08, 2014 9:33 AM
To: ceph-devel@vger.kernel.org
Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Hi,
as mentioned during today's meeting, here are the kernel boot parameters which I found to provide the basis for good performance results:
processor.max_cstate=0
intel_idle.max_cstate=0
I understand these to basically turn off any power saving modes of the CPU; the CPU's we are using are like
Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
At the BIOS level, we
- turn off Hyperthraeding
- turn off Turbo mode (in order ot not leave the specifications)
- turn on frequency floor override
We also assert that
/sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
is set to "performance"
Using above we see a constant frequency at the maximum level allowed by the CPU (except Turbo mode).
Best Regards
Andreas Bluemle
On Wed, 8 Oct 2014 02:51:21 +0200
Mark Nelson <mark.nelson@inktank.com> wrote:
> Hi All,
>
> Just a remind that the weekly performance meeting is on Wednesdays at
> 8AM PST. Same bat time, same bat channel!
>
> Etherpad URL:
> http://pad.ceph.com/p/performance_weekly
>
> To join the Meeting:
> https://bluejeans.com/268261044
>
> To join via Browser:
> https://bluejeans.com/268261044/browser
>
> To join with Lync:
> https://bluejeans.com/268261044/lync
>
>
> To join via Room System:
> Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID:
> 268261044
>
> To join via Phone:
> 1) Dial:
> +1 408 740 7256
> +1 888 240 2560(US Toll Free)
> +1 408 317 9253(Alternate Number)
> (see all numbers - http://bluejeans.com/numbers)
> 2) Enter Conference ID: 268261044
>
> Mark
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org More majordomo
> info at http://vger.kernel.org/majordomo-info.html
>
>
--
Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de
ITXperts GmbH http://www.itxperts.de
Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917
D-81541 Muenchen (Germany) Fax: (+49) 89 89044910
Company details: http://www.itxperts.de/imprint.htm
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
2014-10-08 23:55 ` Paul Von-Stamwitz
@ 2014-10-14 11:22 ` Andreas Bluemle
2014-10-14 13:13 ` Sage Weil
0 siblings, 1 reply; 22+ messages in thread
From: Andreas Bluemle @ 2014-10-14 11:22 UTC (permalink / raw)
To: Paul Von-Stamwitz; +Cc: Stefan Priebe, Somnath Roy, ceph-devel@vger.kernel.org
[-- Attachment #1: Type: text/plain, Size: 5412 bytes --]
Hi,
On Wed, 8 Oct 2014 16:55:38 -0700
Paul Von-Stamwitz <PVonStamwitz@us.fujitsu.com> wrote:
>
> > > Hi,
> > >
> > > as mentioned during today's meeting, here are the kernel boot
> > > parameters
> > which I found to provide the basis for good performance results:
> > >
> > > processor.max_cstate=0
> > > intel_idle.max_cstate=0
> > >
> > > I understand these to basically turn off any power saving modes
> > > of the
> > CPU; the CPU's we are using are like
> > > Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
> > > Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
> > >
> > > At the BIOS level, we
> > > - turn off Hyperthraeding
> > > - turn off Turbo mode (in order ot not leave the specifications)
> > > - turn on frequency floor override
> > >
> > > We also assert that
> > > /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
> > > is set to "performance"
> > >
> > > Using above we see a constant frequency at the maximum level
> > > allowed by
> > the CPU (except Turbo mode).
> >
> > How much performance do we gain by this? Till now i thought it's
> > just 1-3% so i'm still running ondemand govenor plus power savings.
>
> As always, it depends. I saw noticeable increases in some throughput
> tests (though I can't recall the % gain.) More important to me was
> that it made my fio results much more consistent. As we measure
> improvements, these settings remove some of the "system noise".
>
> Best,
> Paul
>
There were two different aspects which showed improvemnt:
- code was executed faster
- thread switching delays were reduced significantly
See the attached grahics. They show processing of a 4 kB write
request: processing at the Pipe::Reader is roughly 200 us in both
pictures, and sth. like 20 us at the OSD::Dispatcher. So there
is not much of a benefit here.
But the delay between the end of the Pipe::Reader and the start
of the OSD::Dispatcher threads reduced really significantly.
(And sorry for the late response)
> >
> > Greets,
> > Stefan
> >
> > > Best Regards
> > >
> > > Andreas Bluemle
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Wed, 8 Oct 2014 02:51:21 +0200
> > > Mark Nelson <mark.nelson@inktank.com> wrote:
> > >
> > >> Hi All,
> > >>
> > >> Just a remind that the weekly performance meeting is on
> > >> Wednesdays at 8AM PST. Same bat time, same bat channel!
> > >>
> > >> Etherpad URL:
> > >> http://pad.ceph.com/p/performance_weekly
> > >>
> > >> To join the Meeting:
> > >> https://bluejeans.com/268261044
> > >>
> > >> To join via Browser:
> > >> https://bluejeans.com/268261044/browser
> > >>
> > >> To join with Lync:
> > >> https://bluejeans.com/268261044/lync
> > >>
> > >>
> > >> To join via Room System:
> > >> Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID:
> > >> 268261044
> > >>
> > >> To join via Phone:
> > >> 1) Dial:
> > >> +1 408 740 7256
> > >> +1 888 240 2560(US Toll Free)
> > >> +1 408 317 9253(Alternate Number)
> > >> (see all numbers - http://bluejeans.com/numbers)
> > >> 2) Enter Conference ID: 268261044
> > >>
> > >> Mark
> > >> --
> > >> To unsubscribe from this list: send the line "unsubscribe
> > >> ceph-devel" in the body of a message to
> > >> majordomo@vger.kernel.org More
> > majordomo
> > >> info at http://vger.kernel.org/majordomo-info.html
> > >>
> > >>
> > >
> > >
> > >
> > > --
> > > Andreas Bluemle
> > > mailto:Andreas.Bluemle@itxperts.de ITXperts
> > > GmbH http://www.itxperts.de Balanstrasse
> > > 73, Geb. 08 Phone: (+49) 89 89044917 D-81541 Muenchen
> > > (Germany) Fax: (+49) 89 89044910
> > >
> > > Company details: http://www.itxperts.de/imprint.htm
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe
> > > ceph-devel" in the body of a message to majordomo@vger.kernel.org
> > > More
> > majordomo
> > > info at http://vger.kernel.org/majordomo-info.html
> > >
> > > ________________________________
> > >
> > > PLEASE NOTE: The information contained in this electronic mail
> > > message is
> > intended only for the use of the designated recipient(s) named
> > above. If the reader of this message is not the intended recipient,
> > you are hereby notified that you have received this message in
> > error and that any review, dissemination, distribution, or copying
> > of this message is strictly prohibited. If you have received this
> > communication in error, please notify the sender by telephone or
> > e-mail (as shown above) immediately and destroy any and all copies
> > of this message in your possession (whether hard copies or
> > electronically stored copies).
> > >
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe
> > > ceph-devel" in the body of a message to majordomo@vger.kernel.org
> > > More
> > majordomo
> > > info at http://vger.kernel.org/majordomo-info.html
> > >
> > --
> > To unsubscribe from this list: send the line "unsubscribe
> > ceph-devel" in the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
--
Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de
ITXperts GmbH http://www.itxperts.de
Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917
D-81541 Muenchen (Germany) Fax: (+49) 89 89044910
Company details: http://www.itxperts.de/imprint.htm
[-- Attachment #2: detail-latency-default.png --]
[-- Type: image/png, Size: 12330 bytes --]
[-- Attachment #3: detail-latency-no-powersave-modes.png --]
[-- Type: image/png, Size: 11065 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
2014-10-14 11:22 ` Andreas Bluemle
@ 2014-10-14 13:13 ` Sage Weil
2014-10-14 14:38 ` Andreas Bluemle
0 siblings, 1 reply; 22+ messages in thread
From: Sage Weil @ 2014-10-14 13:13 UTC (permalink / raw)
To: Andreas Bluemle
Cc: Paul Von-Stamwitz, Stefan Priebe, Somnath Roy,
ceph-devel@vger.kernel.org
On Tue, 14 Oct 2014, Andreas Bluemle wrote:
> Hi,
>
>
> On Wed, 8 Oct 2014 16:55:38 -0700
> Paul Von-Stamwitz <PVonStamwitz@us.fujitsu.com> wrote:
>
> >
> > > > Hi,
> > > >
> > > > as mentioned during today's meeting, here are the kernel boot
> > > > parameters
> > > which I found to provide the basis for good performance results:
> > > >
> > > > processor.max_cstate=0
> > > > intel_idle.max_cstate=0
> > > >
> > > > I understand these to basically turn off any power saving modes
> > > > of the
> > > CPU; the CPU's we are using are like
> > > > Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
> > > > Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
> > > >
> > > > At the BIOS level, we
> > > > - turn off Hyperthraeding
> > > > - turn off Turbo mode (in order ot not leave the specifications)
> > > > - turn on frequency floor override
> > > >
> > > > We also assert that
> > > > /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
> > > > is set to "performance"
> > > >
> > > > Using above we see a constant frequency at the maximum level
> > > > allowed by
> > > the CPU (except Turbo mode).
> > >
> > > How much performance do we gain by this? Till now i thought it's
> > > just 1-3% so i'm still running ondemand govenor plus power savings.
> >
> > As always, it depends. I saw noticeable increases in some throughput
> > tests (though I can't recall the % gain.) More important to me was
> > that it made my fio results much more consistent. As we measure
> > improvements, these settings remove some of the "system noise".
> >
> > Best,
> > Paul
> >
>
> There were two different aspects which showed improvemnt:
> - code was executed faster
> - thread switching delays were reduced significantly
>
> See the attached grahics. They show processing of a 4 kB write
> request: processing at the Pipe::Reader is roughly 200 us in both
> pictures, and sth. like 20 us at the OSD::Dispatcher. So there
> is not much of a benefit here.
>
> But the delay between the end of the Pipe::Reader and the start
> of the OSD::Dispatcher threads reduced really significantly.
This test had a single outstanding IO, right? The question for me is if
this reflect latencies we'd see under a realistic workload, where the are
more IOs in flight and the CPUs aren't likely to be in low power states.
I'm not sure how low the load needs to be before those states kick in and
these latencies start to appear...
sage
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
2014-10-14 13:13 ` Sage Weil
@ 2014-10-14 14:38 ` Andreas Bluemle
[not found] ` <75674D092A819E4189E91166C74CB90D0144A660@shsmsx102.ccr.corp.intel.com>
0 siblings, 1 reply; 22+ messages in thread
From: Andreas Bluemle @ 2014-10-14 14:38 UTC (permalink / raw)
To: Sage Weil
Cc: Paul Von-Stamwitz, Stefan Priebe, Somnath Roy,
ceph-devel@vger.kernel.org
Hi Sage,
[embedded below]
On Tue, 14 Oct 2014 06:13:58 -0700 (PDT)
Sage Weil <sage@newdream.net> wrote:
> On Tue, 14 Oct 2014, Andreas Bluemle wrote:
> > Hi,
> >
> >
> > On Wed, 8 Oct 2014 16:55:38 -0700
> > Paul Von-Stamwitz <PVonStamwitz@us.fujitsu.com> wrote:
> >
> > >
> > > > > Hi,
> > > > >
> > > > > as mentioned during today's meeting, here are the kernel boot
> > > > > parameters
> > > > which I found to provide the basis for good performance results:
> > > > >
> > > > > processor.max_cstate=0
> > > > > intel_idle.max_cstate=0
> > > > >
> > > > > I understand these to basically turn off any power saving
> > > > > modes of the
> > > > CPU; the CPU's we are using are like
> > > > > Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
> > > > > Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
> > > > >
> > > > > At the BIOS level, we
> > > > > - turn off Hyperthraeding
> > > > > - turn off Turbo mode (in order ot not leave the
> > > > > specifications)
> > > > > - turn on frequency floor override
> > > > >
> > > > > We also assert that
> > > > > /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
> > > > > is set to "performance"
> > > > >
> > > > > Using above we see a constant frequency at the maximum level
> > > > > allowed by
> > > > the CPU (except Turbo mode).
> > > >
> > > > How much performance do we gain by this? Till now i thought it's
> > > > just 1-3% so i'm still running ondemand govenor plus power
> > > > savings.
> > >
> > > As always, it depends. I saw noticeable increases in some
> > > throughput tests (though I can't recall the % gain.) More
> > > important to me was that it made my fio results much more
> > > consistent. As we measure improvements, these settings remove
> > > some of the "system noise".
> > >
> > > Best,
> > > Paul
> > >
> >
> > There were two different aspects which showed improvemnt:
> > - code was executed faster
> > - thread switching delays were reduced significantly
> >
> > See the attached grahics. They show processing of a 4 kB write
> > request: processing at the Pipe::Reader is roughly 200 us in both
> > pictures, and sth. like 20 us at the OSD::Dispatcher. So there
> > is not much of a benefit here.
> >
> > But the delay between the end of the Pipe::Reader and the start
> > of the OSD::Dispatcher threads reduced really significantly.
>
> This test had a single outstanding IO, right? The question for me is
> if this reflect latencies we'd see under a realistic workload, where
> the are more IOs in flight and the CPUs aren't likely to be in low
> power states. I'm not sure how low the load needs to be before those
> states kick in and these latencies start to appear...
>
> sage
Yes and no...
Yes: the test was a fio sequential write, 4k per write, with a
single IO in flight.
No: this means that on a given object in the osd file store with the
default size of 4 MByte, 1024 subsequent write requests will hit that
object - and hence the corresponding ceph-osd daemon. So even though
the system as a whole was not very busy, the ceph-osd daemon assigned
to the file object under pressure was fairly busy.
The intention of the test was to eliminate additional latencies
because of queues building up.
What the test shows is the contribution of the various processing
steps within ceph-osd to the overall latency for an individual
write requres when CPU power state related effects have been
eliminated,
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
--
Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de
ITXperts GmbH http://www.itxperts.de
Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917
D-81541 Muenchen (Germany) Fax: (+49) 89 89044910
Company details: http://www.itxperts.de/imprint.htm
^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
[not found] ` <75674D092A819E4189E91166C74CB90D0144A660@shsmsx102.ccr.corp.intel.com>
@ 2014-10-15 2:23 ` Sage Weil
2014-10-15 2:43 ` Somnath Roy
2014-10-15 2:59 ` Shu, Xinxin
0 siblings, 2 replies; 22+ messages in thread
From: Sage Weil @ 2014-10-15 2:23 UTC (permalink / raw)
To: Shu, Xinxin
Cc: Andreas Bluemle, Paul Von-Stamwitz, Stefan Priebe, Somnath Roy,
ceph-devel@vger.kernel.org, Zhang, Jian
On Wed, 15 Oct 2014, Shu, Xinxin wrote:
> Hi all , recently we tested 4K random write performance on our full SSD
> setup (12 x Intel DC3700) , but peak performance is ~23K IOPS, which is
> much lower than hardware capability , with detail latency breakdown , we
> found that most of latency comes from osd queue , we have noticed the
> optimizations on osd queue , and tried latest master on our setup , but
> there is a performance regression , we also checked the qlock and pg
> lock with perf counter, the waiting count and latency are very small,
> the attached pdf shows the details , any suggestion will be appreciated
> ?
I would start by making sure 'osd enable op tracker = false' if it isn't
already.
The other thing to keep in mind is that a lot of the work has enabled
OSD perforamnce to scale as the clients increase. It looks like
your test has a single client. Can you try running 2, 4, 8 clients
and see if the per-OSD throughput goes up?
Digging into the code with a tool like vtune would be extremely helpful, I
think. There is a lot of time spent in do_op (osd prepare and osd queue)
that fujitsu has called out but we haven't narrowed down where the time is
being spent.
sage
>
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Andreas Bluemle
> Sent: Tuesday, October 14, 2014 10:38 PM
> To: Sage Weil
> Cc: Paul Von-Stamwitz; Stefan Priebe; Somnath Roy; ceph-devel@vger.kernel.org
> Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
>
> Hi Sage,
>
> [embedded below]
>
> On Tue, 14 Oct 2014 06:13:58 -0700 (PDT) Sage Weil <sage@newdream.net> wrote:
>
> > On Tue, 14 Oct 2014, Andreas Bluemle wrote:
> > > Hi,
> > >
> > >
> > > On Wed, 8 Oct 2014 16:55:38 -0700
> > > Paul Von-Stamwitz <PVonStamwitz@us.fujitsu.com> wrote:
> > >
> > > >
> > > > > > Hi,
> > > > > >
> > > > > > as mentioned during today's meeting, here are the kernel boot
> > > > > > parameters
> > > > > which I found to provide the basis for good performance results:
> > > > > >
> > > > > > processor.max_cstate=0
> > > > > > intel_idle.max_cstate=0
> > > > > >
> > > > > > I understand these to basically turn off any power saving
> > > > > > modes of the
> > > > > CPU; the CPU's we are using are like
> > > > > > Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
> > > > > > Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
> > > > > >
> > > > > > At the BIOS level, we
> > > > > > - turn off Hyperthraeding
> > > > > > - turn off Turbo mode (in order ot not leave the
> > > > > > specifications)
> > > > > > - turn on frequency floor override
> > > > > >
> > > > > > We also assert that
> > > > > > /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
> > > > > > is set to "performance"
> > > > > >
> > > > > > Using above we see a constant frequency at the maximum level
> > > > > > allowed by
> > > > > the CPU (except Turbo mode).
> > > > >
> > > > > How much performance do we gain by this? Till now i thought it's
> > > > > just 1-3% so i'm still running ondemand govenor plus power
> > > > > savings.
> > > >
> > > > As always, it depends. I saw noticeable increases in some
> > > > throughput tests (though I can't recall the % gain.) More
> > > > important to me was that it made my fio results much more
> > > > consistent. As we measure improvements, these settings remove some
> > > > of the "system noise".
> > > >
> > > > Best,
> > > > Paul
> > > >
> > >
> > > There were two different aspects which showed improvemnt:
> > > - code was executed faster
> > > - thread switching delays were reduced significantly
> > >
> > > See the attached grahics. They show processing of a 4 kB write
> > > request: processing at the Pipe::Reader is roughly 200 us in both
> > > pictures, and sth. like 20 us at the OSD::Dispatcher. So there is
> > > not much of a benefit here.
> > >
> > > But the delay between the end of the Pipe::Reader and the start of
> > > the OSD::Dispatcher threads reduced really significantly.
> >
> > This test had a single outstanding IO, right? The question for me is
> > if this reflect latencies we'd see under a realistic workload, where
> > the are more IOs in flight and the CPUs aren't likely to be in low
> > power states. I'm not sure how low the load needs to be before those
> > states kick in and these latencies start to appear...
> >
> > sage
>
> Yes and no...
>
> Yes: the test was a fio sequential write, 4k per write, with a single IO in flight.
>
> No: this means that on a given object in the osd file store with the default size of 4 MByte, 1024 subsequent write requests will hit that object - and hence the corresponding ceph-osd daemon. So even though the system as a whole was not very busy, the ceph-osd daemon assigned to the file object under pressure was fairly busy.
>
> The intention of the test was to eliminate additional latencies because of queues building up.
>
> What the test shows is the contribution of the various processing steps within ceph-osd to the overall latency for an individual write requres when CPU power state related effects have been eliminated,
>
>
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > in the body of a message to majordomo@vger.kernel.org More majordomo
> > info at http://vger.kernel.org/majordomo-info.html
> >
> >
>
>
>
> --
> Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de
> ITXperts GmbH http://www.itxperts.de
> Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917
> D-81541 Muenchen (Germany) Fax: (+49) 89 89044910
>
> Company details: http://www.itxperts.de/imprint.htm
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
2014-10-15 2:23 ` Sage Weil
@ 2014-10-15 2:43 ` Somnath Roy
2014-10-15 2:59 ` Shu, Xinxin
1 sibling, 0 replies; 22+ messages in thread
From: Somnath Roy @ 2014-10-15 2:43 UTC (permalink / raw)
To: Sage Weil, Shu, Xinxin
Cc: Andreas Bluemle, Paul Von-Stamwitz, Stefan Priebe,
ceph-devel@vger.kernel.org, Zhang, Jian
Sage,
I think they seem to be using 7VM (and thus 7 librbd clients) clients for the test.
XinXin,
You are running 2 OSDS/SSD and that is not recommended . Not sure that has an impact or not. Along with disabling optracker as Sage suggested, you may want to tweak the osd num shards and number of filestore threads to see if it is improving performance.
BTW, each librados client is now ~20% slower (even after rbd_cache = false) and with 7 clients adding those degradation could be significant. One quick check you can do to factor out librbd degradation, is to use firefly librbd/librados combination.
Thanks & Regards
Somnath
-----Original Message-----
From: Sage Weil [mailto:sage@newdream.net]
Sent: Tuesday, October 14, 2014 7:23 PM
To: Shu, Xinxin
Cc: Andreas Bluemle; Paul Von-Stamwitz; Stefan Priebe; Somnath Roy; ceph-devel@vger.kernel.org; Zhang, Jian
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
On Wed, 15 Oct 2014, Shu, Xinxin wrote:
> Hi all , recently we tested 4K random write performance on our full
> SSD setup (12 x Intel DC3700) , but peak performance is ~23K IOPS,
> which is much lower than hardware capability , with detail latency
> breakdown , we found that most of latency comes from osd queue , we
> have noticed the optimizations on osd queue , and tried latest master
> on our setup , but there is a performance regression , we also checked
> the qlock and pg lock with perf counter, the waiting count and latency
> are very small, the attached pdf shows the details , any suggestion
> will be appreciated ?
I would start by making sure 'osd enable op tracker = false' if it isn't already.
The other thing to keep in mind is that a lot of the work has enabled OSD perforamnce to scale as the clients increase. It looks like your test has a single client. Can you try running 2, 4, 8 clients and see if the per-OSD throughput goes up?
Digging into the code with a tool like vtune would be extremely helpful, I think. There is a lot of time spent in do_op (osd prepare and osd queue) that fujitsu has called out but we haven't narrowed down where the time is being spent.
sage
>
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org
> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Andreas Bluemle
> Sent: Tuesday, October 14, 2014 10:38 PM
> To: Sage Weil
> Cc: Paul Von-Stamwitz; Stefan Priebe; Somnath Roy;
> ceph-devel@vger.kernel.org
> Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot
> params
>
> Hi Sage,
>
> [embedded below]
>
> On Tue, 14 Oct 2014 06:13:58 -0700 (PDT) Sage Weil <sage@newdream.net> wrote:
>
> > On Tue, 14 Oct 2014, Andreas Bluemle wrote:
> > > Hi,
> > >
> > >
> > > On Wed, 8 Oct 2014 16:55:38 -0700
> > > Paul Von-Stamwitz <PVonStamwitz@us.fujitsu.com> wrote:
> > >
> > > >
> > > > > > Hi,
> > > > > >
> > > > > > as mentioned during today's meeting, here are the kernel
> > > > > > boot parameters
> > > > > which I found to provide the basis for good performance results:
> > > > > >
> > > > > > processor.max_cstate=0
> > > > > > intel_idle.max_cstate=0
> > > > > >
> > > > > > I understand these to basically turn off any power saving
> > > > > > modes of the
> > > > > CPU; the CPU's we are using are like
> > > > > > Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
> > > > > > Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
> > > > > >
> > > > > > At the BIOS level, we
> > > > > > - turn off Hyperthraeding
> > > > > > - turn off Turbo mode (in order ot not leave the
> > > > > > specifications)
> > > > > > - turn on frequency floor override
> > > > > >
> > > > > > We also assert that
> > > > > > /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
> > > > > > is set to "performance"
> > > > > >
> > > > > > Using above we see a constant frequency at the maximum level
> > > > > > allowed by
> > > > > the CPU (except Turbo mode).
> > > > >
> > > > > How much performance do we gain by this? Till now i thought
> > > > > it's just 1-3% so i'm still running ondemand govenor plus
> > > > > power savings.
> > > >
> > > > As always, it depends. I saw noticeable increases in some
> > > > throughput tests (though I can't recall the % gain.) More
> > > > important to me was that it made my fio results much more
> > > > consistent. As we measure improvements, these settings remove
> > > > some of the "system noise".
> > > >
> > > > Best,
> > > > Paul
> > > >
> > >
> > > There were two different aspects which showed improvemnt:
> > > - code was executed faster
> > > - thread switching delays were reduced significantly
> > >
> > > See the attached grahics. They show processing of a 4 kB write
> > > request: processing at the Pipe::Reader is roughly 200 us in both
> > > pictures, and sth. like 20 us at the OSD::Dispatcher. So there is
> > > not much of a benefit here.
> > >
> > > But the delay between the end of the Pipe::Reader and the start of
> > > the OSD::Dispatcher threads reduced really significantly.
> >
> > This test had a single outstanding IO, right? The question for me
> > is if this reflect latencies we'd see under a realistic workload,
> > where the are more IOs in flight and the CPUs aren't likely to be in
> > low power states. I'm not sure how low the load needs to be before
> > those states kick in and these latencies start to appear...
> >
> > sage
>
> Yes and no...
>
> Yes: the test was a fio sequential write, 4k per write, with a single IO in flight.
>
> No: this means that on a given object in the osd file store with the default size of 4 MByte, 1024 subsequent write requests will hit that object - and hence the corresponding ceph-osd daemon. So even though the system as a whole was not very busy, the ceph-osd daemon assigned to the file object under pressure was fairly busy.
>
> The intention of the test was to eliminate additional latencies because of queues building up.
>
> What the test shows is the contribution of the various processing
> steps within ceph-osd to the overall latency for an individual write
> requres when CPU power state related effects have been eliminated,
>
>
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > in the body of a message to majordomo@vger.kernel.org More majordomo
> > info at http://vger.kernel.org/majordomo-info.html
> >
> >
>
>
>
> --
> Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de
> ITXperts GmbH http://www.itxperts.de
> Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917
> D-81541 Muenchen (Germany) Fax: (+49) 89 89044910
>
> Company details: http://www.itxperts.de/imprint.htm
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org More majordomo
> info at http://vger.kernel.org/majordomo-info.html
>
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
2014-10-15 2:23 ` Sage Weil
2014-10-15 2:43 ` Somnath Roy
@ 2014-10-15 2:59 ` Shu, Xinxin
1 sibling, 0 replies; 22+ messages in thread
From: Shu, Xinxin @ 2014-10-15 2:59 UTC (permalink / raw)
To: Sage Weil
Cc: Andreas Bluemle, Paul Von-Stamwitz, Stefan Priebe, Somnath Roy,
ceph-devel@vger.kernel.org, Zhang, Jian
Hi sage ,
With latest master , we do set 'osd_enable_op_tracker = false' , we tested up to 7 rbd clients in our test , but after two clients, the iops is stable at ~23K, there is no performance gain with more clients
-----Original Message-----
From: Sage Weil [mailto:sage@newdream.net]
Sent: Wednesday, October 15, 2014 10:23 AM
To: Shu, Xinxin
Cc: Andreas Bluemle; Paul Von-Stamwitz; Stefan Priebe; Somnath Roy; ceph-devel@vger.kernel.org; Zhang, Jian
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
On Wed, 15 Oct 2014, Shu, Xinxin wrote:
> Hi all , recently we tested 4K random write performance on our full
> SSD setup (12 x Intel DC3700) , but peak performance is ~23K IOPS,
> which is much lower than hardware capability , with detail latency
> breakdown , we found that most of latency comes from osd queue , we
> have noticed the optimizations on osd queue , and tried latest master
> on our setup , but there is a performance regression , we also checked
> the qlock and pg lock with perf counter, the waiting count and latency
> are very small, the attached pdf shows the details , any suggestion
> will be appreciated ?
I would start by making sure 'osd enable op tracker = false' if it isn't already.
The other thing to keep in mind is that a lot of the work has enabled OSD perforamnce to scale as the clients increase. It looks like your test has a single client. Can you try running 2, 4, 8 clients and see if the per-OSD throughput goes up?
Digging into the code with a tool like vtune would be extremely helpful, I think. There is a lot of time spent in do_op (osd prepare and osd queue) that fujitsu has called out but we haven't narrowed down where the time is being spent.
sage
>
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org
> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Andreas Bluemle
> Sent: Tuesday, October 14, 2014 10:38 PM
> To: Sage Weil
> Cc: Paul Von-Stamwitz; Stefan Priebe; Somnath Roy;
> ceph-devel@vger.kernel.org
> Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot
> params
>
> Hi Sage,
>
> [embedded below]
>
> On Tue, 14 Oct 2014 06:13:58 -0700 (PDT) Sage Weil <sage@newdream.net> wrote:
>
> > On Tue, 14 Oct 2014, Andreas Bluemle wrote:
> > > Hi,
> > >
> > >
> > > On Wed, 8 Oct 2014 16:55:38 -0700
> > > Paul Von-Stamwitz <PVonStamwitz@us.fujitsu.com> wrote:
> > >
> > > >
> > > > > > Hi,
> > > > > >
> > > > > > as mentioned during today's meeting, here are the kernel
> > > > > > boot parameters
> > > > > which I found to provide the basis for good performance results:
> > > > > >
> > > > > > processor.max_cstate=0
> > > > > > intel_idle.max_cstate=0
> > > > > >
> > > > > > I understand these to basically turn off any power saving
> > > > > > modes of the
> > > > > CPU; the CPU's we are using are like
> > > > > > Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
> > > > > > Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
> > > > > >
> > > > > > At the BIOS level, we
> > > > > > - turn off Hyperthraeding
> > > > > > - turn off Turbo mode (in order ot not leave the
> > > > > > specifications)
> > > > > > - turn on frequency floor override
> > > > > >
> > > > > > We also assert that
> > > > > > /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
> > > > > > is set to "performance"
> > > > > >
> > > > > > Using above we see a constant frequency at the maximum level
> > > > > > allowed by
> > > > > the CPU (except Turbo mode).
> > > > >
> > > > > How much performance do we gain by this? Till now i thought
> > > > > it's just 1-3% so i'm still running ondemand govenor plus
> > > > > power savings.
> > > >
> > > > As always, it depends. I saw noticeable increases in some
> > > > throughput tests (though I can't recall the % gain.) More
> > > > important to me was that it made my fio results much more
> > > > consistent. As we measure improvements, these settings remove
> > > > some of the "system noise".
> > > >
> > > > Best,
> > > > Paul
> > > >
> > >
> > > There were two different aspects which showed improvemnt:
> > > - code was executed faster
> > > - thread switching delays were reduced significantly
> > >
> > > See the attached grahics. They show processing of a 4 kB write
> > > request: processing at the Pipe::Reader is roughly 200 us in both
> > > pictures, and sth. like 20 us at the OSD::Dispatcher. So there is
> > > not much of a benefit here.
> > >
> > > But the delay between the end of the Pipe::Reader and the start of
> > > the OSD::Dispatcher threads reduced really significantly.
> >
> > This test had a single outstanding IO, right? The question for me
> > is if this reflect latencies we'd see under a realistic workload,
> > where the are more IOs in flight and the CPUs aren't likely to be in
> > low power states. I'm not sure how low the load needs to be before
> > those states kick in and these latencies start to appear...
> >
> > sage
>
> Yes and no...
>
> Yes: the test was a fio sequential write, 4k per write, with a single IO in flight.
>
> No: this means that on a given object in the osd file store with the default size of 4 MByte, 1024 subsequent write requests will hit that object - and hence the corresponding ceph-osd daemon. So even though the system as a whole was not very busy, the ceph-osd daemon assigned to the file object under pressure was fairly busy.
>
> The intention of the test was to eliminate additional latencies because of queues building up.
>
> What the test shows is the contribution of the various processing
> steps within ceph-osd to the overall latency for an individual write
> requres when CPU power state related effects have been eliminated,
>
>
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > in the body of a message to majordomo@vger.kernel.org More majordomo
> > info at http://vger.kernel.org/majordomo-info.html
> >
> >
>
>
>
> --
> Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de
> ITXperts GmbH http://www.itxperts.de
> Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917
> D-81541 Muenchen (Germany) Fax: (+49) 89 89044910
>
> Company details: http://www.itxperts.de/imprint.htm
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org More majordomo
> info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
2014-10-09 0:50 ` Somnath Roy
2014-10-09 1:07 ` Mark Nelson
2014-10-10 23:39 ` Duan, Jiangang
@ 2014-11-05 14:33 ` Zhang, Jian
2 siblings, 0 replies; 22+ messages in thread
From: Zhang, Jian @ 2014-11-05 14:33 UTC (permalink / raw)
To: 'Somnath Roy', Duan, Jiangang, Andreas Bluemle,
ceph-devel@vger.kernel.org
Cc: Zhang, Jian
Somnath,
Recently we completed some test regarding on HT ON & OFF, my conclusion is HT has little impact on performance and latency.
Test configuration:
* Cluster and RBD node Config:
2 OSD nodes: 2x E5-2680 v2 @ 2.80GHz, 32GB Memory, 8x S350 480G SSD connected via LSI9205 HBA, 10GB NIC
2 Client nodes: 2x E5-2580 v2 @2.80GHz, 32GB memory, 10GB NIC
Ceph: v0.86
Workloads: fio+rbd engine, 64K_Rand_Read_qd64, created 8 Volumes, 60G each
* Performance
HT_Disabled:
-----------------
IOPS: 29298
BW (MB/s): 1831
Average RT(msec): 558
HT_Enabled:
-------------------
IOPS: 29308
BW(MB/s):1832
Average RT(msec): 558
-------------------
We also found some old data we did a while back for 4K random read on another full ssd setup based on cuttlefish, running fio inslide virtual machine with qemu-rbd driver,
the conclusion remains the same.
Ceph: 192.168.10.45,op_threads=32,pg=16384,HT_enable,total 4 osds,
Throughput Latency
HT_ON HT_OFF HT_ON/OFF HT_ON HT_OFF HT_ON/OFF
1vm 22615.80 22483.40 101% 2.54 2.50 102%
2vm 45094.60 45050.30 100% 2.50 2.47 101%
3vm 67636.60 68884.50 98% 2.96 2.88 103%
4vm 67560.60 68198.00 99% 4.09 4.04 101%
5vm 65893.30 67044.40 98% 5.25 5.13 102%
6vm 64446.50 65331.40 99% 6.41 6.31 102%
7vm 63305.02 64345.33 98% 7.57 7.44 102%
8vm 62840.17 63060.58 100% 8.66 8.62 101%
Let me know if you have other questions.
Thanks
Jian
-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Somnath Roy
Sent: Thursday, October 09, 2014 8:51 AM
To: Duan, Jiangang; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Hi Jiangang,
I managed to get some data for you but it's for a 3 node cluster. I will try to get data for single node as well.
Test config:
-------------
Cluster and rbd node config:
----------------------------------
"2x E5-2680 10C 2.8GHz 25M
8x 16GB RDIMM, dual rank x4 (128GB)
Mellanox MT27500 40 Gigabit Ethernet
LSI 9207 SAS HBA"
8 X 800 GB SSDs (Optimus Eco) per cluster node
3 cluster nodes + 3 rbd nodes
Total storage ~ 19 TB
We have total 24 OSDs running , each node has 8 OSDs/SSD
Configured 3 pools with 528 PGs/pool and 6 RBDs/pool . Each RBD image size is ~230G.
We have tried on 64K_RR_QD64 workload here.
HT_ENABLE
--------------
IOPS : 112500
Throughput (MB/S): 7012
Avg Resp.Time (m.sec): 17
Max Resp.Time (m.sec): 3184
HT_DISABLE
--------------
IOPS : 120864
Throughput (MB/S): 7530
Avg Resp.Time (m.sec): 11
Max Resp.Time (m.sec): 1056
So, ~7% iop increase but response time decrease is ~35% which is real good.
Thanks & Regards
Somnath
-----Original Message-----
From: Duan, Jiangang [mailto:jiangang.duan@intel.com]
Sent: Wednesday, October 08, 2014 1:03 PM
To: Somnath Roy; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Sound good. Thanks. -jiangang
-----Original Message-----
From: Somnath Roy [mailto:Somnath.Roy@sandisk.com]
Sent: Wednesday, October 08, 2014 10:53 AM
To: Duan, Jiangang; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Hi Jiangang,
Give me a day or two, I will gather all the data and share with community.
Thanks & Regards
Somnath
-----Original Message-----
From: Duan, Jiangang [mailto:jiangang.duan@intel.com]
Sent: Wednesday, October 08, 2014 10:47 AM
To: Somnath Roy; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Can you guys share the w/ HT and w/o HT data? I want to take a look at that to understand why.
-jiangang
-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Somnath Roy
Sent: Wednesday, October 08, 2014 10:38 AM
To: Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Thanks Andres for sharing this. I will try those out.
BTW, I am using Ubuntu 14.04 LTS and couldn't find any sysfs entry like 'cpufreq'..
root@stormeap-4:~# ll /sys/devices/system/cpu/cpu10/
cache/ crash_notes driver/ microcode/ online subsystem/ topology/
cpuidle/ crash_notes_size firmware_node/ node0/ power/ thermal_throttle/ uevent
I am using Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz.
Regards
Somnath
-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Andreas Bluemle
Sent: Wednesday, October 08, 2014 9:33 AM
To: ceph-devel@vger.kernel.org
Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Hi,
as mentioned during today's meeting, here are the kernel boot parameters which I found to provide the basis for good performance results:
processor.max_cstate=0
intel_idle.max_cstate=0
I understand these to basically turn off any power saving modes of the CPU; the CPU's we are using are like
Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
At the BIOS level, we
- turn off Hyperthraeding
- turn off Turbo mode (in order ot not leave the specifications)
- turn on frequency floor override
We also assert that
/sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
is set to "performance"
Using above we see a constant frequency at the maximum level allowed by the CPU (except Turbo mode).
Best Regards
Andreas Bluemle
On Wed, 8 Oct 2014 02:51:21 +0200
Mark Nelson <mark.nelson@inktank.com> wrote:
> Hi All,
>
> Just a remind that the weekly performance meeting is on Wednesdays at
> 8AM PST. Same bat time, same bat channel!
>
> Etherpad URL:
> http://pad.ceph.com/p/performance_weekly
>
> To join the Meeting:
> https://bluejeans.com/268261044
>
> To join via Browser:
> https://bluejeans.com/268261044/browser
>
> To join with Lync:
> https://bluejeans.com/268261044/lync
>
>
> To join via Room System:
> Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID:
> 268261044
>
> To join via Phone:
> 1) Dial:
> +1 408 740 7256
> +1 888 240 2560(US Toll Free)
> +1 408 317 9253(Alternate Number)
> (see all numbers - http://bluejeans.com/numbers)
> 2) Enter Conference ID: 268261044
>
> Mark
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org More majordomo
> info at http://vger.kernel.org/majordomo-info.html
>
>
--
Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de
ITXperts GmbH http://www.itxperts.de
Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917
D-81541 Muenchen (Germany) Fax: (+49) 89 89044910
Company details: http://www.itxperts.de/imprint.htm
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 22+ messages in thread
end of thread, other threads:[~2014-11-05 14:35 UTC | newest]
Thread overview: 22+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-10-08 0:51 10/7/2014 Weekly Ceph Performance Meeting Mark Nelson
2014-10-08 16:32 ` 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params Andreas Bluemle
2014-10-08 17:38 ` Somnath Roy
2014-10-08 17:47 ` Duan, Jiangang
2014-10-08 17:53 ` Somnath Roy
2014-10-08 20:03 ` Duan, Jiangang
2014-10-09 0:50 ` Somnath Roy
2014-10-09 1:07 ` Mark Nelson
2014-10-09 6:45 ` Somnath Roy
2014-10-10 23:39 ` Duan, Jiangang
2014-10-10 23:43 ` Somnath Roy
2014-11-05 14:33 ` Zhang, Jian
2014-10-08 17:57 ` Loic Dachary
2014-10-08 18:07 ` Alexandre DERUMIER
2014-10-08 18:35 ` Stefan Priebe
2014-10-08 23:55 ` Paul Von-Stamwitz
2014-10-14 11:22 ` Andreas Bluemle
2014-10-14 13:13 ` Sage Weil
2014-10-14 14:38 ` Andreas Bluemle
[not found] ` <75674D092A819E4189E91166C74CB90D0144A660@shsmsx102.ccr.corp.intel.com>
2014-10-15 2:23 ` Sage Weil
2014-10-15 2:43 ` Somnath Roy
2014-10-15 2:59 ` Shu, Xinxin
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.