Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

[Mark Nelson <mark.nelson@inktank.com>]

Hi Somnath,

Was this with HT enabled/disabled on both the cluster and the RBD nodes?

Mark

On 10/08/2014 07:50 PM, Somnath Roy wrote:
> Hi Jiangang,
> I managed to get some data for you but it's for a 3 node cluster. I will try to get data for single node as well.
>
> Test config:
> -------------
>
> Cluster and rbd node config:
> ----------------------------------
> "2x E5-2680 10C 2.8GHz 25M
> 8x 16GB RDIMM, dual rank x4 (128GB)
> Mellanox MT27500 40 Gigabit Ethernet
> LSI 9207 SAS HBA"
>
> 8 X 800 GB SSDs (Optimus Eco) per cluster node
>
> 3 cluster nodes + 3 rbd nodes
>
> Total storage ~ 19 TB
>
> We have total 24 OSDs running , each node has 8 OSDs/SSD
>
> Configured 3 pools with 528 PGs/pool and 6 RBDs/pool . Each RBD image size is ~230G.
>
> We have tried on 64K_RR_QD64 workload here.
>
> HT_ENABLE
> --------------
>
> IOPS :  				112500
> Throughput (MB/S):		7012
> Avg Resp.Time (m.sec):                17
> Max Resp.Time (m.sec):              3184
>
> HT_DISABLE
> --------------
>
> IOPS :  				120864
> Throughput (MB/S):		7530
> Avg Resp.Time (m.sec):                11
> Max Resp.Time (m.sec):              1056
>
>
> So, ~7% iop increase but response time decrease is ~35% which is real good.
>
> Thanks & Regards
> Somnath
>
> -----Original Message-----
> From: Duan, Jiangang [mailto:jiangang.duan@intel.com]
> Sent: Wednesday, October 08, 2014 1:03 PM
> To: Somnath Roy; Andreas Bluemle; ceph-devel@vger.kernel.org
> Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
>
> Sound good. Thanks. -jiangang
>
> -----Original Message-----
> From: Somnath Roy [mailto:Somnath.Roy@sandisk.com]
> Sent: Wednesday, October 08, 2014 10:53 AM
> To: Duan, Jiangang; Andreas Bluemle; ceph-devel@vger.kernel.org
> Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
>
> Hi Jiangang,
> Give me a day or two, I will gather all the data and share with community.
>
> Thanks & Regards
> Somnath
>
> -----Original Message-----
> From: Duan, Jiangang [mailto:jiangang.duan@intel.com]
> Sent: Wednesday, October 08, 2014 10:47 AM
> To: Somnath Roy; Andreas Bluemle; ceph-devel@vger.kernel.org
> Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
>
> Can you guys share the w/ HT and w/o HT data? I want to take a look at that to understand why.
>
> -jiangang
>
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Somnath Roy
> Sent: Wednesday, October 08, 2014 10:38 AM
> To: Andreas Bluemle; ceph-devel@vger.kernel.org
> Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
>
> Thanks Andres for sharing this. I will try those out.
> BTW, I am using Ubuntu 14.04 LTS and couldn't find any sysfs entry like  'cpufreq'..
>
> root@stormeap-4:~# ll /sys/devices/system/cpu/cpu10/
> cache/            crash_notes       driver/           microcode/        online            subsystem/        topology/
> cpuidle/          crash_notes_size  firmware_node/    node0/            power/            thermal_throttle/ uevent
>
> I am using Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz.
>
> Regards
> Somnath
>
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Andreas Bluemle
> Sent: Wednesday, October 08, 2014 9:33 AM
> To: ceph-devel@vger.kernel.org
> Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
>
> Hi,
>
> as mentioned during today's meeting, here are the kernel boot parameters which I found to provide the basis for good performance results:
>
>    processor.max_cstate=0
>    intel_idle.max_cstate=0
>
> I understand these to basically turn off any power saving modes of the CPU; the CPU's we are using are like
>   Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
>   Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
>
> At the BIOS level, we
>   - turn off Hyperthraeding
>   - turn off Turbo mode (in order ot not leave the specifications)
>   - turn on frequency floor override
>
> We also assert that
>   /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
>   is set to "performance"
>
> Using above we see a constant frequency at the maximum level allowed by the CPU (except Turbo mode).
>
>
> Best Regards
>
> Andreas Bluemle
>
>
>
>
>
>
> On Wed, 8 Oct 2014 02:51:21 +0200
> Mark Nelson <mark.nelson@inktank.com> wrote:
>
>> Hi All,
>>
>> Just a remind that the weekly performance meeting is on Wednesdays at
>> 8AM PST.  Same bat time, same bat channel!
>>
>> Etherpad URL:
>> http://pad.ceph.com/p/performance_weekly
>>
>> To join the Meeting:
>> https://bluejeans.com/268261044
>>
>> To join via Browser:
>> https://bluejeans.com/268261044/browser
>>
>> To join with Lync:
>> https://bluejeans.com/268261044/lync
>>
>>
>> To join via Room System:
>> Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID:
>> 268261044
>>
>> To join via Phone:
>> 1) Dial:
>>             +1 408 740 7256
>>             +1 888 240 2560(US Toll Free)
>>             +1 408 317 9253(Alternate Number)
>>             (see all numbers - http://bluejeans.com/numbers)
>> 2) Enter Conference ID: 268261044
>>
>> Mark
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>> in the body of a message to majordomo@vger.kernel.org More majordomo
>> info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>
>
>
> --
> Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
> ITXperts GmbH                       http://www.itxperts.de
> Balanstrasse 73, Geb. 08            Phone: (+49) 89 89044917
> D-81541 Muenchen (Germany)          Fax:   (+49) 89 89044910
>
> Company details: http://www.itxperts.de/imprint.htm
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> ________________________________
>
> PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>