10/7/2014 Weekly Ceph Performance Meeting

10/7/2014 Weekly Ceph Performance Meeting

[Mark Nelson]

Hi All,

Just a remind that the weekly performance meeting is on Wednesdays at 
8AM PST.  Same bat time, same bat channel!

Etherpad URL:
http://pad.ceph.com/p/performance_weekly

To join the Meeting:
https://bluejeans.com/268261044

To join via Browser:
https://bluejeans.com/268261044/browser

To join with Lync:
https://bluejeans.com/268261044/lync


To join via Room System:
Video Conferencing System: bjn.vc -or- 199.48.152.152
Meeting ID: 268261044

To join via Phone:
1) Dial:
           +1 408 740 7256
           +1 888 240 2560(US Toll Free)
           +1 408 317 9253(Alternate Number)
           (see all numbers - http://bluejeans.com/numbers)
2) Enter Conference ID: 268261044

Mark

Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

[Andreas Bluemle]

Hi,

as mentioned during today's meeting, here are the kernel
boot parameters which I found to provide the basis for
good performance results:

  processor.max_cstate=0
  intel_idle.max_cstate=0

I understand these to basically turn off any power saving modes
of the CPU; the CPU's we are using are like
 Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
 Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz

At the BIOS level, we 
 - turn off Hyperthraeding
 - turn off Turbo mode (in order ot not leave the specifications)
 - turn on frequency floor override

We also assert that
 /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
 is set to "performance"

Using above we see a constant frequency at the maximum level
allowed by the CPU (except Turbo mode).


Best Regards

Andreas Bluemle






On Wed, 8 Oct 2014 02:51:21 +0200
Mark Nelson <mark.nelson@inktank.com> wrote:

> Hi All,
> 
> Just a remind that the weekly performance meeting is on Wednesdays at 
> 8AM PST.  Same bat time, same bat channel!
> 
> Etherpad URL:
> http://pad.ceph.com/p/performance_weekly
> 
> To join the Meeting:
> https://bluejeans.com/268261044
> 
> To join via Browser:
> https://bluejeans.com/268261044/browser
> 
> To join with Lync:
> https://bluejeans.com/268261044/lync
> 
> 
> To join via Room System:
> Video Conferencing System: bjn.vc -or- 199.48.152.152
> Meeting ID: 268261044
> 
> To join via Phone:
> 1) Dial:
>            +1 408 740 7256
>            +1 888 240 2560(US Toll Free)
>            +1 408 317 9253(Alternate Number)
>            (see all numbers - http://bluejeans.com/numbers)
> 2) Enter Conference ID: 268261044
> 
> Mark
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 



-- 
Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
ITXperts GmbH                       http://www.itxperts.de
Balanstrasse 73, Geb. 08            Phone: (+49) 89 89044917
D-81541 Muenchen (Germany)          Fax:   (+49) 89 89044910

Company details: http://www.itxperts.de/imprint.htm

RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

[Somnath Roy]

Thanks Andres for sharing this. I will try those out.
BTW, I am using Ubuntu 14.04 LTS and couldn't find any sysfs entry like  'cpufreq'..

root@stormeap-4:~# ll /sys/devices/system/cpu/cpu10/
cache/            crash_notes       driver/           microcode/        online            subsystem/        topology/
cpuidle/          crash_notes_size  firmware_node/    node0/            power/            thermal_throttle/ uevent

I am using Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz.

Regards
Somnath

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Andreas Bluemle
Sent: Wednesday, October 08, 2014 9:33 AM
To: ceph-devel@vger.kernel.org
Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Hi,

as mentioned during today's meeting, here are the kernel boot parameters which I found to provide the basis for good performance results:

  processor.max_cstate=0
  intel_idle.max_cstate=0

I understand these to basically turn off any power saving modes of the CPU; the CPU's we are using are like
 Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
 Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz

At the BIOS level, we
 - turn off Hyperthraeding
 - turn off Turbo mode (in order ot not leave the specifications)
 - turn on frequency floor override

We also assert that
 /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
 is set to "performance"

Using above we see a constant frequency at the maximum level allowed by the CPU (except Turbo mode).


Best Regards

Andreas Bluemle






On Wed, 8 Oct 2014 02:51:21 +0200
Mark Nelson <mark.nelson@inktank.com> wrote:

> Hi All,
>
> Just a remind that the weekly performance meeting is on Wednesdays at
> 8AM PST.  Same bat time, same bat channel!
>
> Etherpad URL:
> http://pad.ceph.com/p/performance_weekly
>
> To join the Meeting:
> https://bluejeans.com/268261044
>
> To join via Browser:
> https://bluejeans.com/268261044/browser
>
> To join with Lync:
> https://bluejeans.com/268261044/lync
>
>
> To join via Room System:
> Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID:
> 268261044
>
> To join via Phone:
> 1) Dial:
>            +1 408 740 7256
>            +1 888 240 2560(US Toll Free)
>            +1 408 317 9253(Alternate Number)
>            (see all numbers - http://bluejeans.com/numbers)
> 2) Enter Conference ID: 268261044
>
> Mark
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org More majordomo
> info at  http://vger.kernel.org/majordomo-info.html
>
>



--
Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
ITXperts GmbH                       http://www.itxperts.de
Balanstrasse 73, Geb. 08            Phone: (+49) 89 89044917
D-81541 Muenchen (Germany)          Fax:   (+49) 89 89044910

Company details: http://www.itxperts.de/imprint.htm
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

[Duan, Jiangang]

Can you guys share the w/ HT and w/o HT data? I want to take a look at that to understand why. 

-jiangang

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Somnath Roy
Sent: Wednesday, October 08, 2014 10:38 AM
To: Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Thanks Andres for sharing this. I will try those out.
BTW, I am using Ubuntu 14.04 LTS and couldn't find any sysfs entry like  'cpufreq'..

root@stormeap-4:~# ll /sys/devices/system/cpu/cpu10/
cache/            crash_notes       driver/           microcode/        online            subsystem/        topology/
cpuidle/          crash_notes_size  firmware_node/    node0/            power/            thermal_throttle/ uevent

I am using Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz.

Regards
Somnath

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Andreas Bluemle
Sent: Wednesday, October 08, 2014 9:33 AM
To: ceph-devel@vger.kernel.org
Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Hi,

as mentioned during today's meeting, here are the kernel boot parameters which I found to provide the basis for good performance results:

  processor.max_cstate=0
  intel_idle.max_cstate=0

I understand these to basically turn off any power saving modes of the CPU; the CPU's we are using are like
 Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
 Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz

At the BIOS level, we
 - turn off Hyperthraeding
 - turn off Turbo mode (in order ot not leave the specifications)
 - turn on frequency floor override

We also assert that
 /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
 is set to "performance"

Using above we see a constant frequency at the maximum level allowed by the CPU (except Turbo mode).


Best Regards

Andreas Bluemle






On Wed, 8 Oct 2014 02:51:21 +0200
Mark Nelson <mark.nelson@inktank.com> wrote:

> Hi All,
>
> Just a remind that the weekly performance meeting is on Wednesdays at 
> 8AM PST.  Same bat time, same bat channel!
>
> Etherpad URL:
> http://pad.ceph.com/p/performance_weekly
>
> To join the Meeting:
> https://bluejeans.com/268261044
>
> To join via Browser:
> https://bluejeans.com/268261044/browser
>
> To join with Lync:
> https://bluejeans.com/268261044/lync
>
>
> To join via Room System:
> Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID:
> 268261044
>
> To join via Phone:
> 1) Dial:
>            +1 408 740 7256
>            +1 888 240 2560(US Toll Free)
>            +1 408 317 9253(Alternate Number)
>            (see all numbers - http://bluejeans.com/numbers)
> 2) Enter Conference ID: 268261044
>
> Mark
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html
>
>



--
Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
ITXperts GmbH                       http://www.itxperts.de
Balanstrasse 73, Geb. 08            Phone: (+49) 89 89044917
D-81541 Muenchen (Germany)          Fax:   (+49) 89 89044910

Company details: http://www.itxperts.de/imprint.htm
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

[Somnath Roy]

Hi Jiangang,
Give me a day or two, I will gather all the data and share with community.

Thanks & Regards
Somnath

-----Original Message-----
From: Duan, Jiangang [mailto:jiangang.duan@intel.com] 
Sent: Wednesday, October 08, 2014 10:47 AM
To: Somnath Roy; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Can you guys share the w/ HT and w/o HT data? I want to take a look at that to understand why. 

-jiangang

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Somnath Roy
Sent: Wednesday, October 08, 2014 10:38 AM
To: Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Thanks Andres for sharing this. I will try those out.
BTW, I am using Ubuntu 14.04 LTS and couldn't find any sysfs entry like  'cpufreq'..

root@stormeap-4:~# ll /sys/devices/system/cpu/cpu10/
cache/            crash_notes       driver/           microcode/        online            subsystem/        topology/
cpuidle/          crash_notes_size  firmware_node/    node0/            power/            thermal_throttle/ uevent

I am using Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz.

Regards
Somnath

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Andreas Bluemle
Sent: Wednesday, October 08, 2014 9:33 AM
To: ceph-devel@vger.kernel.org
Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Hi,

as mentioned during today's meeting, here are the kernel boot parameters which I found to provide the basis for good performance results:

  processor.max_cstate=0
  intel_idle.max_cstate=0

I understand these to basically turn off any power saving modes of the CPU; the CPU's we are using are like
 Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
 Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz

At the BIOS level, we
 - turn off Hyperthraeding
 - turn off Turbo mode (in order ot not leave the specifications)
 - turn on frequency floor override

We also assert that
 /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
 is set to "performance"

Using above we see a constant frequency at the maximum level allowed by the CPU (except Turbo mode).


Best Regards

Andreas Bluemle






On Wed, 8 Oct 2014 02:51:21 +0200
Mark Nelson <mark.nelson@inktank.com> wrote:

> Hi All,
>
> Just a remind that the weekly performance meeting is on Wednesdays at 
> 8AM PST.  Same bat time, same bat channel!
>
> Etherpad URL:
> http://pad.ceph.com/p/performance_weekly
>
> To join the Meeting:
> https://bluejeans.com/268261044
>
> To join via Browser:
> https://bluejeans.com/268261044/browser
>
> To join with Lync:
> https://bluejeans.com/268261044/lync
>
>
> To join via Room System:
> Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID:
> 268261044
>
> To join via Phone:
> 1) Dial:
>            +1 408 740 7256
>            +1 888 240 2560(US Toll Free)
>            +1 408 317 9253(Alternate Number)
>            (see all numbers - http://bluejeans.com/numbers)
> 2) Enter Conference ID: 268261044
>
> Mark
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html
>
>



--
Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
ITXperts GmbH                       http://www.itxperts.de
Balanstrasse 73, Geb. 08            Phone: (+49) 89 89044917
D-81541 Muenchen (Germany)          Fax:   (+49) 89 89044910

Company details: http://www.itxperts.de/imprint.htm
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

[Duan, Jiangang]

Sound good. Thanks. -jiangang

-----Original Message-----
From: Somnath Roy [mailto:Somnath.Roy@sandisk.com] 
Sent: Wednesday, October 08, 2014 10:53 AM
To: Duan, Jiangang; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Hi Jiangang,
Give me a day or two, I will gather all the data and share with community.

Thanks & Regards
Somnath

-----Original Message-----
From: Duan, Jiangang [mailto:jiangang.duan@intel.com]
Sent: Wednesday, October 08, 2014 10:47 AM
To: Somnath Roy; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Can you guys share the w/ HT and w/o HT data? I want to take a look at that to understand why. 

-jiangang

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Somnath Roy
Sent: Wednesday, October 08, 2014 10:38 AM
To: Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Thanks Andres for sharing this. I will try those out.
BTW, I am using Ubuntu 14.04 LTS and couldn't find any sysfs entry like  'cpufreq'..

root@stormeap-4:~# ll /sys/devices/system/cpu/cpu10/
cache/            crash_notes       driver/           microcode/        online            subsystem/        topology/
cpuidle/          crash_notes_size  firmware_node/    node0/            power/            thermal_throttle/ uevent

I am using Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz.

Regards
Somnath

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Andreas Bluemle
Sent: Wednesday, October 08, 2014 9:33 AM
To: ceph-devel@vger.kernel.org
Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Hi,

as mentioned during today's meeting, here are the kernel boot parameters which I found to provide the basis for good performance results:

  processor.max_cstate=0
  intel_idle.max_cstate=0

I understand these to basically turn off any power saving modes of the CPU; the CPU's we are using are like
 Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
 Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz

At the BIOS level, we
 - turn off Hyperthraeding
 - turn off Turbo mode (in order ot not leave the specifications)
 - turn on frequency floor override

We also assert that
 /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
 is set to "performance"

Using above we see a constant frequency at the maximum level allowed by the CPU (except Turbo mode).


Best Regards

Andreas Bluemle






On Wed, 8 Oct 2014 02:51:21 +0200
Mark Nelson <mark.nelson@inktank.com> wrote:

> Hi All,
>
> Just a remind that the weekly performance meeting is on Wednesdays at 
> 8AM PST.  Same bat time, same bat channel!
>
> Etherpad URL:
> http://pad.ceph.com/p/performance_weekly
>
> To join the Meeting:
> https://bluejeans.com/268261044
>
> To join via Browser:
> https://bluejeans.com/268261044/browser
>
> To join with Lync:
> https://bluejeans.com/268261044/lync
>
>
> To join via Room System:
> Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID:
> 268261044
>
> To join via Phone:
> 1) Dial:
>            +1 408 740 7256
>            +1 888 240 2560(US Toll Free)
>            +1 408 317 9253(Alternate Number)
>            (see all numbers - http://bluejeans.com/numbers)
> 2) Enter Conference ID: 268261044
>
> Mark
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html
>
>



--
Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
ITXperts GmbH                       http://www.itxperts.de
Balanstrasse 73, Geb. 08            Phone: (+49) 89 89044917
D-81541 Muenchen (Germany)          Fax:   (+49) 89 89044910

Company details: http://www.itxperts.de/imprint.htm
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

[Somnath Roy]

Hi Jiangang,
I managed to get some data for you but it's for a 3 node cluster. I will try to get data for single node as well.

Test config:
-------------

Cluster and rbd node config:
----------------------------------
"2x E5-2680 10C 2.8GHz 25M
8x 16GB RDIMM, dual rank x4 (128GB)
Mellanox MT27500 40 Gigabit Ethernet
LSI 9207 SAS HBA"

8 X 800 GB SSDs (Optimus Eco) per cluster node

3 cluster nodes + 3 rbd nodes 

Total storage ~ 19 TB

We have total 24 OSDs running , each node has 8 OSDs/SSD

Configured 3 pools with 528 PGs/pool and 6 RBDs/pool . Each RBD image size is ~230G.

We have tried on 64K_RR_QD64 workload here.

HT_ENABLE
--------------

IOPS :  				112500 
Throughput (MB/S):		7012
Avg Resp.Time (m.sec):                17
Max Resp.Time (m.sec):              3184

HT_DISABLE
--------------

IOPS :  				120864 
Throughput (MB/S):		7530
Avg Resp.Time (m.sec):                11
Max Resp.Time (m.sec):              1056


So, ~7% iop increase but response time decrease is ~35% which is real good.

Thanks & Regards
Somnath

-----Original Message-----
From: Duan, Jiangang [mailto:jiangang.duan@intel.com] 
Sent: Wednesday, October 08, 2014 1:03 PM
To: Somnath Roy; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Sound good. Thanks. -jiangang

-----Original Message-----
From: Somnath Roy [mailto:Somnath.Roy@sandisk.com]
Sent: Wednesday, October 08, 2014 10:53 AM
To: Duan, Jiangang; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Hi Jiangang,
Give me a day or two, I will gather all the data and share with community.

Thanks & Regards
Somnath

-----Original Message-----
From: Duan, Jiangang [mailto:jiangang.duan@intel.com]
Sent: Wednesday, October 08, 2014 10:47 AM
To: Somnath Roy; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Can you guys share the w/ HT and w/o HT data? I want to take a look at that to understand why. 

-jiangang

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Somnath Roy
Sent: Wednesday, October 08, 2014 10:38 AM
To: Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Thanks Andres for sharing this. I will try those out.
BTW, I am using Ubuntu 14.04 LTS and couldn't find any sysfs entry like  'cpufreq'..

root@stormeap-4:~# ll /sys/devices/system/cpu/cpu10/
cache/            crash_notes       driver/           microcode/        online            subsystem/        topology/
cpuidle/          crash_notes_size  firmware_node/    node0/            power/            thermal_throttle/ uevent

I am using Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz.

Regards
Somnath

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Andreas Bluemle
Sent: Wednesday, October 08, 2014 9:33 AM
To: ceph-devel@vger.kernel.org
Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Hi,

as mentioned during today's meeting, here are the kernel boot parameters which I found to provide the basis for good performance results:

  processor.max_cstate=0
  intel_idle.max_cstate=0

I understand these to basically turn off any power saving modes of the CPU; the CPU's we are using are like
 Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
 Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz

At the BIOS level, we
 - turn off Hyperthraeding
 - turn off Turbo mode (in order ot not leave the specifications)
 - turn on frequency floor override

We also assert that
 /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
 is set to "performance"

Using above we see a constant frequency at the maximum level allowed by the CPU (except Turbo mode).


Best Regards

Andreas Bluemle






On Wed, 8 Oct 2014 02:51:21 +0200
Mark Nelson <mark.nelson@inktank.com> wrote:

> Hi All,
>
> Just a remind that the weekly performance meeting is on Wednesdays at 
> 8AM PST.  Same bat time, same bat channel!
>
> Etherpad URL:
> http://pad.ceph.com/p/performance_weekly
>
> To join the Meeting:
> https://bluejeans.com/268261044
>
> To join via Browser:
> https://bluejeans.com/268261044/browser
>
> To join with Lync:
> https://bluejeans.com/268261044/lync
>
>
> To join via Room System:
> Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID:
> 268261044
>
> To join via Phone:
> 1) Dial:
>            +1 408 740 7256
>            +1 888 240 2560(US Toll Free)
>            +1 408 317 9253(Alternate Number)
>            (see all numbers - http://bluejeans.com/numbers)
> 2) Enter Conference ID: 268261044
>
> Mark
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html
>
>



--
Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
ITXperts GmbH                       http://www.itxperts.de
Balanstrasse 73, Geb. 08            Phone: (+49) 89 89044917
D-81541 Muenchen (Germany)          Fax:   (+49) 89 89044910

Company details: http://www.itxperts.de/imprint.htm
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

[Mark Nelson]

Hi Somnath,

Was this with HT enabled/disabled on both the cluster and the RBD nodes?

Mark

On 10/08/2014 07:50 PM, Somnath Roy wrote:
> Hi Jiangang,
> I managed to get some data for you but it's for a 3 node cluster. I will try to get data for single node as well.
>
> Test config:
> -------------
>
> Cluster and rbd node config:
> ----------------------------------
> "2x E5-2680 10C 2.8GHz 25M
> 8x 16GB RDIMM, dual rank x4 (128GB)
> Mellanox MT27500 40 Gigabit Ethernet
> LSI 9207 SAS HBA"
>
> 8 X 800 GB SSDs (Optimus Eco) per cluster node
>
> 3 cluster nodes + 3 rbd nodes
>
> Total storage ~ 19 TB
>
> We have total 24 OSDs running , each node has 8 OSDs/SSD
>
> Configured 3 pools with 528 PGs/pool and 6 RBDs/pool . Each RBD image size is ~230G.
>
> We have tried on 64K_RR_QD64 workload here.
>
> HT_ENABLE
> --------------
>
> IOPS :  				112500
> Throughput (MB/S):		7012
> Avg Resp.Time (m.sec):                17
> Max Resp.Time (m.sec):              3184
>
> HT_DISABLE
> --------------
>
> IOPS :  				120864
> Throughput (MB/S):		7530
> Avg Resp.Time (m.sec):                11
> Max Resp.Time (m.sec):              1056
>
>
> So, ~7% iop increase but response time decrease is ~35% which is real good.
>
> Thanks & Regards
> Somnath
>
> -----Original Message-----
> From: Duan, Jiangang [mailto:jiangang.duan@intel.com]
> Sent: Wednesday, October 08, 2014 1:03 PM
> To: Somnath Roy; Andreas Bluemle; ceph-devel@vger.kernel.org
> Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
>
> Sound good. Thanks. -jiangang
>
> -----Original Message-----
> From: Somnath Roy [mailto:Somnath.Roy@sandisk.com]
> Sent: Wednesday, October 08, 2014 10:53 AM
> To: Duan, Jiangang; Andreas Bluemle; ceph-devel@vger.kernel.org
> Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
>
> Hi Jiangang,
> Give me a day or two, I will gather all the data and share with community.
>
> Thanks & Regards
> Somnath
>
> -----Original Message-----
> From: Duan, Jiangang [mailto:jiangang.duan@intel.com]
> Sent: Wednesday, October 08, 2014 10:47 AM
> To: Somnath Roy; Andreas Bluemle; ceph-devel@vger.kernel.org
> Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
>
> Can you guys share the w/ HT and w/o HT data? I want to take a look at that to understand why.
>
> -jiangang
>
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Somnath Roy
> Sent: Wednesday, October 08, 2014 10:38 AM
> To: Andreas Bluemle; ceph-devel@vger.kernel.org
> Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
>
> Thanks Andres for sharing this. I will try those out.
> BTW, I am using Ubuntu 14.04 LTS and couldn't find any sysfs entry like  'cpufreq'..
>
> root@stormeap-4:~# ll /sys/devices/system/cpu/cpu10/
> cache/            crash_notes       driver/           microcode/        online            subsystem/        topology/
> cpuidle/          crash_notes_size  firmware_node/    node0/            power/            thermal_throttle/ uevent
>
> I am using Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz.
>
> Regards
> Somnath
>
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Andreas Bluemle
> Sent: Wednesday, October 08, 2014 9:33 AM
> To: ceph-devel@vger.kernel.org
> Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
>
> Hi,
>
> as mentioned during today's meeting, here are the kernel boot parameters which I found to provide the basis for good performance results:
>
>    processor.max_cstate=0
>    intel_idle.max_cstate=0
>
> I understand these to basically turn off any power saving modes of the CPU; the CPU's we are using are like
>   Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
>   Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
>
> At the BIOS level, we
>   - turn off Hyperthraeding
>   - turn off Turbo mode (in order ot not leave the specifications)
>   - turn on frequency floor override
>
> We also assert that
>   /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
>   is set to "performance"
>
> Using above we see a constant frequency at the maximum level allowed by the CPU (except Turbo mode).
>
>
> Best Regards
>
> Andreas Bluemle
>
>
>
>
>
>
> On Wed, 8 Oct 2014 02:51:21 +0200
> Mark Nelson <mark.nelson@inktank.com> wrote:
>
>> Hi All,
>>
>> Just a remind that the weekly performance meeting is on Wednesdays at
>> 8AM PST.  Same bat time, same bat channel!
>>
>> Etherpad URL:
>> http://pad.ceph.com/p/performance_weekly
>>
>> To join the Meeting:
>> https://bluejeans.com/268261044
>>
>> To join via Browser:
>> https://bluejeans.com/268261044/browser
>>
>> To join with Lync:
>> https://bluejeans.com/268261044/lync
>>
>>
>> To join via Room System:
>> Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID:
>> 268261044
>>
>> To join via Phone:
>> 1) Dial:
>>             +1 408 740 7256
>>             +1 888 240 2560(US Toll Free)
>>             +1 408 317 9253(Alternate Number)
>>             (see all numbers - http://bluejeans.com/numbers)
>> 2) Enter Conference ID: 268261044
>>
>> Mark
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>> in the body of a message to majordomo@vger.kernel.org More majordomo
>> info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>
>
>
> --
> Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
> ITXperts GmbH                       http://www.itxperts.de
> Balanstrasse 73, Geb. 08            Phone: (+49) 89 89044917
> D-81541 Muenchen (Germany)          Fax:   (+49) 89 89044910
>
> Company details: http://www.itxperts.de/imprint.htm
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> ________________________________
>
> PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

[Somnath Roy]

Yes, Mark...

-----Original Message-----
From: Mark Nelson [mailto:mark.nelson@inktank.com] 
Sent: Wednesday, October 08, 2014 6:08 PM
To: Somnath Roy; Duan, Jiangang; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Hi Somnath,

Was this with HT enabled/disabled on both the cluster and the RBD nodes?

Mark

On 10/08/2014 07:50 PM, Somnath Roy wrote:
> Hi Jiangang,
> I managed to get some data for you but it's for a 3 node cluster. I will try to get data for single node as well.
>
> Test config:
> -------------
>
> Cluster and rbd node config:
> ----------------------------------
> "2x E5-2680 10C 2.8GHz 25M
> 8x 16GB RDIMM, dual rank x4 (128GB)
> Mellanox MT27500 40 Gigabit Ethernet
> LSI 9207 SAS HBA"
>
> 8 X 800 GB SSDs (Optimus Eco) per cluster node
>
> 3 cluster nodes + 3 rbd nodes
>
> Total storage ~ 19 TB
>
> We have total 24 OSDs running , each node has 8 OSDs/SSD
>
> Configured 3 pools with 528 PGs/pool and 6 RBDs/pool . Each RBD image size is ~230G.
>
> We have tried on 64K_RR_QD64 workload here.
>
> HT_ENABLE
> --------------
>
> IOPS :  				112500
> Throughput (MB/S):		7012
> Avg Resp.Time (m.sec):                17
> Max Resp.Time (m.sec):              3184
>
> HT_DISABLE
> --------------
>
> IOPS :  				120864
> Throughput (MB/S):		7530
> Avg Resp.Time (m.sec):                11
> Max Resp.Time (m.sec):              1056
>
>
> So, ~7% iop increase but response time decrease is ~35% which is real good.
>
> Thanks & Regards
> Somnath
>
> -----Original Message-----
> From: Duan, Jiangang [mailto:jiangang.duan@intel.com]
> Sent: Wednesday, October 08, 2014 1:03 PM
> To: Somnath Roy; Andreas Bluemle; ceph-devel@vger.kernel.org
> Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot 
> params
>
> Sound good. Thanks. -jiangang
>
> -----Original Message-----
> From: Somnath Roy [mailto:Somnath.Roy@sandisk.com]
> Sent: Wednesday, October 08, 2014 10:53 AM
> To: Duan, Jiangang; Andreas Bluemle; ceph-devel@vger.kernel.org
> Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot 
> params
>
> Hi Jiangang,
> Give me a day or two, I will gather all the data and share with community.
>
> Thanks & Regards
> Somnath
>
> -----Original Message-----
> From: Duan, Jiangang [mailto:jiangang.duan@intel.com]
> Sent: Wednesday, October 08, 2014 10:47 AM
> To: Somnath Roy; Andreas Bluemle; ceph-devel@vger.kernel.org
> Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot 
> params
>
> Can you guys share the w/ HT and w/o HT data? I want to take a look at that to understand why.
>
> -jiangang
>
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org 
> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Somnath Roy
> Sent: Wednesday, October 08, 2014 10:38 AM
> To: Andreas Bluemle; ceph-devel@vger.kernel.org
> Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot 
> params
>
> Thanks Andres for sharing this. I will try those out.
> BTW, I am using Ubuntu 14.04 LTS and couldn't find any sysfs entry like  'cpufreq'..
>
> root@stormeap-4:~# ll /sys/devices/system/cpu/cpu10/
> cache/            crash_notes       driver/           microcode/        online            subsystem/        topology/
> cpuidle/          crash_notes_size  firmware_node/    node0/            power/            thermal_throttle/ uevent
>
> I am using Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz.
>
> Regards
> Somnath
>
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org 
> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Andreas Bluemle
> Sent: Wednesday, October 08, 2014 9:33 AM
> To: ceph-devel@vger.kernel.org
> Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot 
> params
>
> Hi,
>
> as mentioned during today's meeting, here are the kernel boot parameters which I found to provide the basis for good performance results:
>
>    processor.max_cstate=0
>    intel_idle.max_cstate=0
>
> I understand these to basically turn off any power saving modes of the CPU; the CPU's we are using are like
>   Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
>   Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
>
> At the BIOS level, we
>   - turn off Hyperthraeding
>   - turn off Turbo mode (in order ot not leave the specifications)
>   - turn on frequency floor override
>
> We also assert that
>   /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
>   is set to "performance"
>
> Using above we see a constant frequency at the maximum level allowed by the CPU (except Turbo mode).
>
>
> Best Regards
>
> Andreas Bluemle
>
>
>
>
>
>
> On Wed, 8 Oct 2014 02:51:21 +0200
> Mark Nelson <mark.nelson@inktank.com> wrote:
>
>> Hi All,
>>
>> Just a remind that the weekly performance meeting is on Wednesdays at 
>> 8AM PST.  Same bat time, same bat channel!
>>
>> Etherpad URL:
>> http://pad.ceph.com/p/performance_weekly
>>
>> To join the Meeting:
>> https://bluejeans.com/268261044
>>
>> To join via Browser:
>> https://bluejeans.com/268261044/browser
>>
>> To join with Lync:
>> https://bluejeans.com/268261044/lync
>>
>>
>> To join via Room System:
>> Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID:
>> 268261044
>>
>> To join via Phone:
>> 1) Dial:
>>             +1 408 740 7256
>>             +1 888 240 2560(US Toll Free)
>>             +1 408 317 9253(Alternate Number)
>>             (see all numbers - http://bluejeans.com/numbers)
>> 2) Enter Conference ID: 268261044
>>
>> Mark
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>> in the body of a message to majordomo@vger.kernel.org More majordomo 
>> info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>
>
>
> --
> Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
> ITXperts GmbH                       http://www.itxperts.de
> Balanstrasse 73, Geb. 08            Phone: (+49) 89 89044917
> D-81541 Muenchen (Germany)          Fax:   (+49) 89 89044910
>
> Company details: http://www.itxperts.de/imprint.htm
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html
>
> ________________________________
>
> PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html
>

RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

[Duan, Jiangang]

Thanks. let's try do this test on our setup.
BTW, what workload you use here?

-jiangang

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Somnath Roy
Sent: Wednesday, October 08, 2014 5:51 PM
To: Duan, Jiangang; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Hi Jiangang,
I managed to get some data for you but it's for a 3 node cluster. I will try to get data for single node as well.

Test config:
-------------

Cluster and rbd node config:
----------------------------------
"2x E5-2680 10C 2.8GHz 25M
8x 16GB RDIMM, dual rank x4 (128GB)
Mellanox MT27500 40 Gigabit Ethernet
LSI 9207 SAS HBA"

8 X 800 GB SSDs (Optimus Eco) per cluster node

3 cluster nodes + 3 rbd nodes 

Total storage ~ 19 TB

We have total 24 OSDs running , each node has 8 OSDs/SSD

Configured 3 pools with 528 PGs/pool and 6 RBDs/pool . Each RBD image size is ~230G.

We have tried on 64K_RR_QD64 workload here.

HT_ENABLE
--------------

IOPS :  				112500 
Throughput (MB/S):		7012
Avg Resp.Time (m.sec):                17
Max Resp.Time (m.sec):              3184

HT_DISABLE
--------------

IOPS :  				120864 
Throughput (MB/S):		7530
Avg Resp.Time (m.sec):                11
Max Resp.Time (m.sec):              1056


So, ~7% iop increase but response time decrease is ~35% which is real good.

Thanks & Regards
Somnath

-----Original Message-----
From: Duan, Jiangang [mailto:jiangang.duan@intel.com]
Sent: Wednesday, October 08, 2014 1:03 PM
To: Somnath Roy; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Sound good. Thanks. -jiangang

-----Original Message-----
From: Somnath Roy [mailto:Somnath.Roy@sandisk.com]
Sent: Wednesday, October 08, 2014 10:53 AM
To: Duan, Jiangang; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Hi Jiangang,
Give me a day or two, I will gather all the data and share with community.

Thanks & Regards
Somnath

-----Original Message-----
From: Duan, Jiangang [mailto:jiangang.duan@intel.com]
Sent: Wednesday, October 08, 2014 10:47 AM
To: Somnath Roy; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Can you guys share the w/ HT and w/o HT data? I want to take a look at that to understand why. 

-jiangang

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Somnath Roy
Sent: Wednesday, October 08, 2014 10:38 AM
To: Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Thanks Andres for sharing this. I will try those out.
BTW, I am using Ubuntu 14.04 LTS and couldn't find any sysfs entry like  'cpufreq'..

root@stormeap-4:~# ll /sys/devices/system/cpu/cpu10/
cache/            crash_notes       driver/           microcode/        online            subsystem/        topology/
cpuidle/          crash_notes_size  firmware_node/    node0/            power/            thermal_throttle/ uevent

I am using Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz.

Regards
Somnath

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Andreas Bluemle
Sent: Wednesday, October 08, 2014 9:33 AM
To: ceph-devel@vger.kernel.org
Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Hi,

as mentioned during today's meeting, here are the kernel boot parameters which I found to provide the basis for good performance results:

  processor.max_cstate=0
  intel_idle.max_cstate=0

I understand these to basically turn off any power saving modes of the CPU; the CPU's we are using are like
 Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
 Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz

At the BIOS level, we
 - turn off Hyperthraeding
 - turn off Turbo mode (in order ot not leave the specifications)
 - turn on frequency floor override

We also assert that
 /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
 is set to "performance"

Using above we see a constant frequency at the maximum level allowed by the CPU (except Turbo mode).


Best Regards

Andreas Bluemle






On Wed, 8 Oct 2014 02:51:21 +0200
Mark Nelson <mark.nelson@inktank.com> wrote:

> Hi All,
>
> Just a remind that the weekly performance meeting is on Wednesdays at 
> 8AM PST.  Same bat time, same bat channel!
>
> Etherpad URL:
> http://pad.ceph.com/p/performance_weekly
>
> To join the Meeting:
> https://bluejeans.com/268261044
>
> To join via Browser:
> https://bluejeans.com/268261044/browser
>
> To join with Lync:
> https://bluejeans.com/268261044/lync
>
>
> To join via Room System:
> Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID:
> 268261044
>
> To join via Phone:
> 1) Dial:
>            +1 408 740 7256
>            +1 888 240 2560(US Toll Free)
>            +1 408 317 9253(Alternate Number)
>            (see all numbers - http://bluejeans.com/numbers)
> 2) Enter Conference ID: 268261044
>
> Mark
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html
>
>



--
Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
ITXperts GmbH                       http://www.itxperts.de
Balanstrasse 73, Geb. 08            Phone: (+49) 89 89044917
D-81541 Muenchen (Germany)          Fax:   (+49) 89 89044910

Company details: http://www.itxperts.de/imprint.htm
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

[Somnath Roy]

As I mentioned, total workload is ~19 TB...Each RBD is ~230 GB and io_size = 64K..

-----Original Message-----
From: Duan, Jiangang [mailto:jiangang.duan@intel.com] 
Sent: Friday, October 10, 2014 4:39 PM
To: Somnath Roy; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Thanks. let's try do this test on our setup.
BTW, what workload you use here?

-jiangang

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Somnath Roy
Sent: Wednesday, October 08, 2014 5:51 PM
To: Duan, Jiangang; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Hi Jiangang,
I managed to get some data for you but it's for a 3 node cluster. I will try to get data for single node as well.

Test config:
-------------

Cluster and rbd node config:
----------------------------------
"2x E5-2680 10C 2.8GHz 25M
8x 16GB RDIMM, dual rank x4 (128GB)
Mellanox MT27500 40 Gigabit Ethernet
LSI 9207 SAS HBA"

8 X 800 GB SSDs (Optimus Eco) per cluster node

3 cluster nodes + 3 rbd nodes 

Total storage ~ 19 TB

We have total 24 OSDs running , each node has 8 OSDs/SSD

Configured 3 pools with 528 PGs/pool and 6 RBDs/pool . Each RBD image size is ~230G.

We have tried on 64K_RR_QD64 workload here.

HT_ENABLE
--------------

IOPS :  				112500 
Throughput (MB/S):		7012
Avg Resp.Time (m.sec):                17
Max Resp.Time (m.sec):              3184

HT_DISABLE
--------------

IOPS :  				120864 
Throughput (MB/S):		7530
Avg Resp.Time (m.sec):                11
Max Resp.Time (m.sec):              1056


So, ~7% iop increase but response time decrease is ~35% which is real good.

Thanks & Regards
Somnath

-----Original Message-----
From: Duan, Jiangang [mailto:jiangang.duan@intel.com]
Sent: Wednesday, October 08, 2014 1:03 PM
To: Somnath Roy; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Sound good. Thanks. -jiangang

-----Original Message-----
From: Somnath Roy [mailto:Somnath.Roy@sandisk.com]
Sent: Wednesday, October 08, 2014 10:53 AM
To: Duan, Jiangang; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Hi Jiangang,
Give me a day or two, I will gather all the data and share with community.

Thanks & Regards
Somnath

-----Original Message-----
From: Duan, Jiangang [mailto:jiangang.duan@intel.com]
Sent: Wednesday, October 08, 2014 10:47 AM
To: Somnath Roy; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Can you guys share the w/ HT and w/o HT data? I want to take a look at that to understand why. 

-jiangang

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Somnath Roy
Sent: Wednesday, October 08, 2014 10:38 AM
To: Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Thanks Andres for sharing this. I will try those out.
BTW, I am using Ubuntu 14.04 LTS and couldn't find any sysfs entry like  'cpufreq'..

root@stormeap-4:~# ll /sys/devices/system/cpu/cpu10/
cache/            crash_notes       driver/           microcode/        online            subsystem/        topology/
cpuidle/          crash_notes_size  firmware_node/    node0/            power/            thermal_throttle/ uevent

I am using Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz.

Regards
Somnath

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Andreas Bluemle
Sent: Wednesday, October 08, 2014 9:33 AM
To: ceph-devel@vger.kernel.org
Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Hi,

as mentioned during today's meeting, here are the kernel boot parameters which I found to provide the basis for good performance results:

  processor.max_cstate=0
  intel_idle.max_cstate=0

I understand these to basically turn off any power saving modes of the CPU; the CPU's we are using are like
 Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
 Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz

At the BIOS level, we
 - turn off Hyperthraeding
 - turn off Turbo mode (in order ot not leave the specifications)
 - turn on frequency floor override

We also assert that
 /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
 is set to "performance"

Using above we see a constant frequency at the maximum level allowed by the CPU (except Turbo mode).


Best Regards

Andreas Bluemle






On Wed, 8 Oct 2014 02:51:21 +0200
Mark Nelson <mark.nelson@inktank.com> wrote:

> Hi All,
>
> Just a remind that the weekly performance meeting is on Wednesdays at 
> 8AM PST.  Same bat time, same bat channel!
>
> Etherpad URL:
> http://pad.ceph.com/p/performance_weekly
>
> To join the Meeting:
> https://bluejeans.com/268261044
>
> To join via Browser:
> https://bluejeans.com/268261044/browser
>
> To join with Lync:
> https://bluejeans.com/268261044/lync
>
>
> To join via Room System:
> Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID:
> 268261044
>
> To join via Phone:
> 1) Dial:
>            +1 408 740 7256
>            +1 888 240 2560(US Toll Free)
>            +1 408 317 9253(Alternate Number)
>            (see all numbers - http://bluejeans.com/numbers)
> 2) Enter Conference ID: 268261044
>
> Mark
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html
>
>



--
Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
ITXperts GmbH                       http://www.itxperts.de
Balanstrasse 73, Geb. 08            Phone: (+49) 89 89044917
D-81541 Muenchen (Germany)          Fax:   (+49) 89 89044910

Company details: http://www.itxperts.de/imprint.htm
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

[Zhang, Jian]

Somnath,
Recently we completed some test regarding on HT ON & OFF, my conclusion is HT has little impact on performance and latency. 

Test configuration: 

* Cluster and RBD node Config: 
    2 OSD nodes: 2x E5-2680 v2 @ 2.80GHz, 32GB Memory, 8x S350 480G SSD connected via LSI9205 HBA, 10GB NIC		
    2 Client nodes: 2x E5-2580 v2 @2.80GHz, 32GB memory, 10GB NIC
    Ceph: v0.86
    Workloads:  	fio+rbd engine, 64K_Rand_Read_qd64, created 8 Volumes, 60G each
* Performance
HT_Disabled:
-----------------
IOPS: 29298
BW (MB/s): 1831
Average RT(msec): 558

HT_Enabled: 
-------------------
IOPS: 29308
BW(MB/s):1832
Average RT(msec): 558

-------------------
We also found some old data we did a while back for 4K random read on another full ssd setup based on cuttlefish, running fio inslide virtual machine with qemu-rbd driver, 
the conclusion remains the same. 

Ceph: 192.168.10.45,op_threads=32,pg=16384,HT_enable,total 4 osds,									
									
    	Throughput			               Latency					
	HT_ON	    	HT_OFF	HT_ON/OFF	   	 HT_ON	HT_OFF	HT_ON/OFF			
1vm	22615.80 	22483.40 	101%	    		2.54 	2.50 	102%			
2vm	45094.60 	45050.30 	100%	    		2.50 	2.47 	101%			
3vm	67636.60 	68884.50 	98%	       		 2.96 	2.88 	103%			
4vm	67560.60 	68198.00 	99%			4.09 	4.04 	101%			
5vm	65893.30 	67044.40 	98%			5.25 	5.13 	102%			
6vm	64446.50 	65331.40 	99%			6.41 	6.31 	102%			
7vm	63305.02 	64345.33 	98%			7.57 	7.44 	102%			
8vm	62840.17 	63060.58 	100%			8.66 	8.62 	101%			

Let me know if you have other questions. 

Thanks
Jian

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Somnath Roy
Sent: Thursday, October 09, 2014 8:51 AM
To: Duan, Jiangang; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Hi Jiangang,
I managed to get some data for you but it's for a 3 node cluster. I will try to get data for single node as well.

Test config:
-------------

Cluster and rbd node config:
----------------------------------
"2x E5-2680 10C 2.8GHz 25M
8x 16GB RDIMM, dual rank x4 (128GB)
Mellanox MT27500 40 Gigabit Ethernet
LSI 9207 SAS HBA"

8 X 800 GB SSDs (Optimus Eco) per cluster node

3 cluster nodes + 3 rbd nodes 

Total storage ~ 19 TB

We have total 24 OSDs running , each node has 8 OSDs/SSD

Configured 3 pools with 528 PGs/pool and 6 RBDs/pool . Each RBD image size is ~230G.

We have tried on 64K_RR_QD64 workload here.

HT_ENABLE
--------------

IOPS :  				112500 
Throughput (MB/S):		7012
Avg Resp.Time (m.sec):                17
Max Resp.Time (m.sec):              3184

HT_DISABLE
--------------

IOPS :  				120864 
Throughput (MB/S):		7530
Avg Resp.Time (m.sec):                11
Max Resp.Time (m.sec):              1056


So, ~7% iop increase but response time decrease is ~35% which is real good.

Thanks & Regards
Somnath

-----Original Message-----
From: Duan, Jiangang [mailto:jiangang.duan@intel.com]
Sent: Wednesday, October 08, 2014 1:03 PM
To: Somnath Roy; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Sound good. Thanks. -jiangang

-----Original Message-----
From: Somnath Roy [mailto:Somnath.Roy@sandisk.com]
Sent: Wednesday, October 08, 2014 10:53 AM
To: Duan, Jiangang; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Hi Jiangang,
Give me a day or two, I will gather all the data and share with community.

Thanks & Regards
Somnath

-----Original Message-----
From: Duan, Jiangang [mailto:jiangang.duan@intel.com]
Sent: Wednesday, October 08, 2014 10:47 AM
To: Somnath Roy; Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Can you guys share the w/ HT and w/o HT data? I want to take a look at that to understand why. 

-jiangang

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Somnath Roy
Sent: Wednesday, October 08, 2014 10:38 AM
To: Andreas Bluemle; ceph-devel@vger.kernel.org
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Thanks Andres for sharing this. I will try those out.
BTW, I am using Ubuntu 14.04 LTS and couldn't find any sysfs entry like  'cpufreq'..

root@stormeap-4:~# ll /sys/devices/system/cpu/cpu10/
cache/            crash_notes       driver/           microcode/        online            subsystem/        topology/
cpuidle/          crash_notes_size  firmware_node/    node0/            power/            thermal_throttle/ uevent

I am using Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz.

Regards
Somnath

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Andreas Bluemle
Sent: Wednesday, October 08, 2014 9:33 AM
To: ceph-devel@vger.kernel.org
Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

Hi,

as mentioned during today's meeting, here are the kernel boot parameters which I found to provide the basis for good performance results:

  processor.max_cstate=0
  intel_idle.max_cstate=0

I understand these to basically turn off any power saving modes of the CPU; the CPU's we are using are like
 Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
 Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz

At the BIOS level, we
 - turn off Hyperthraeding
 - turn off Turbo mode (in order ot not leave the specifications)
 - turn on frequency floor override

We also assert that
 /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
 is set to "performance"

Using above we see a constant frequency at the maximum level allowed by the CPU (except Turbo mode).


Best Regards

Andreas Bluemle






On Wed, 8 Oct 2014 02:51:21 +0200
Mark Nelson <mark.nelson@inktank.com> wrote:

> Hi All,
>
> Just a remind that the weekly performance meeting is on Wednesdays at 
> 8AM PST.  Same bat time, same bat channel!
>
> Etherpad URL:
> http://pad.ceph.com/p/performance_weekly
>
> To join the Meeting:
> https://bluejeans.com/268261044
>
> To join via Browser:
> https://bluejeans.com/268261044/browser
>
> To join with Lync:
> https://bluejeans.com/268261044/lync
>
>
> To join via Room System:
> Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID:
> 268261044
>
> To join via Phone:
> 1) Dial:
>            +1 408 740 7256
>            +1 888 240 2560(US Toll Free)
>            +1 408 317 9253(Alternate Number)
>            (see all numbers - http://bluejeans.com/numbers)
> 2) Enter Conference ID: 268261044
>
> Mark
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html
>
>



--
Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
ITXperts GmbH                       http://www.itxperts.de
Balanstrasse 73, Geb. 08            Phone: (+49) 89 89044917
D-81541 Muenchen (Germany)          Fax:   (+49) 89 89044910

Company details: http://www.itxperts.de/imprint.htm
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 5287 bytes --]

Hi,

On 08/10/2014 19:38, Somnath Roy wrote:
> Thanks Andres for sharing this. I will try those out.
> BTW, I am using Ubuntu 14.04 LTS and couldn't find any sysfs entry like  'cpufreq'..
> 
> root@stormeap-4:~# ll /sys/devices/system/cpu/cpu10/
> cache/            crash_notes       driver/           microcode/        online            subsystem/        topology/
> cpuidle/          crash_notes_size  firmware_node/    node0/            power/            thermal_throttle/ uevent
> 
> I am using Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz.


-rw-r--r-- 1 root root 4096 oct.   8 17:31 /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
-rw-r--r-- 1 root root 4096 oct.   8 17:31 /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor
-rw-r--r-- 1 root root 4096 oct.   8 17:31 /sys/devices/system/cpu/cpu2/cpufreq/scaling_governor
-rw-r--r-- 1 root root 4096 oct.   8 17:31 /sys/devices/system/cpu/cpu3/cpufreq/scaling_governor
-rw-r--r-- 1 root root 4096 oct.   8 17:31 /sys/devices/system/cpu/cpu4/cpufreq/scaling_governor
-rw-r--r-- 1 root root 4096 oct.   8 17:31 /sys/devices/system/cpu/cpu5/cpufreq/scaling_governor
-rw-r--r-- 1 root root 4096 oct.   8 17:31 /sys/devices/system/cpu/cpu6/cpufreq/scaling_governor
-rw-r--r-- 1 root root 4096 oct.   8 17:31 /sys/devices/system/cpu/cpu7/cpufreq/scaling_governor

model name	: Intel(R) Core(TM) i7-4900MQ CPU @ 2.80GHz

$ lsb_release -d
Description:	Ubuntu Trusty Tahr (development branch)

Cheers

> 
> Regards
> Somnath
> 
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Andreas Bluemle
> Sent: Wednesday, October 08, 2014 9:33 AM
> To: ceph-devel@vger.kernel.org
> Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
> 
> Hi,
> 
> as mentioned during today's meeting, here are the kernel boot parameters which I found to provide the basis for good performance results:
> 
>   processor.max_cstate=0
>   intel_idle.max_cstate=0
> 
> I understand these to basically turn off any power saving modes of the CPU; the CPU's we are using are like
>  Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
>  Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
> 
> At the BIOS level, we
>  - turn off Hyperthraeding
>  - turn off Turbo mode (in order ot not leave the specifications)
>  - turn on frequency floor override
> 
> We also assert that
>  /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
>  is set to "performance"
> 
> Using above we see a constant frequency at the maximum level allowed by the CPU (except Turbo mode).
> 
> 
> Best Regards
> 
> Andreas Bluemle
> 
> 
> 
> 
> 
> 
> On Wed, 8 Oct 2014 02:51:21 +0200
> Mark Nelson <mark.nelson@inktank.com> wrote:
> 
>> Hi All,
>>
>> Just a remind that the weekly performance meeting is on Wednesdays at
>> 8AM PST.  Same bat time, same bat channel!
>>
>> Etherpad URL:
>> http://pad.ceph.com/p/performance_weekly
>>
>> To join the Meeting:
>> https://bluejeans.com/268261044
>>
>> To join via Browser:
>> https://bluejeans.com/268261044/browser
>>
>> To join with Lync:
>> https://bluejeans.com/268261044/lync
>>
>>
>> To join via Room System:
>> Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID:
>> 268261044
>>
>> To join via Phone:
>> 1) Dial:
>>            +1 408 740 7256
>>            +1 888 240 2560(US Toll Free)
>>            +1 408 317 9253(Alternate Number)
>>            (see all numbers - http://bluejeans.com/numbers)
>> 2) Enter Conference ID: 268261044
>>
>> Mark
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>> in the body of a message to majordomo@vger.kernel.org More majordomo
>> info at  http://vger.kernel.org/majordomo-info.html
>>
>>
> 
> 
> 
> --
> Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
> ITXperts GmbH                       http://www.itxperts.de
> Balanstrasse 73, Geb. 08            Phone: (+49) 89 89044917
> D-81541 Muenchen (Germany)          Fax:   (+49) 89 89044910
> 
> Company details: http://www.itxperts.de/imprint.htm
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> ________________________________
> 
> PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

[Alexandre DERUMIER]

hi,

>>BTW, I am using Ubuntu 14.04 LTS and couldn't find any sysfs entry like 'cpufreq'.. 

check this arch wiki about kernel modules needed
https://wiki.archlinux.org/index.php/CPU_frequency_scaling



Also note that all theses tuning can normaly be done at bios level.
(on last dell servers bios, setting power profile to max performance, is setting the governor to max and disable all cstate)

I'm always doing it on my kvm hypervisors hosts.


They are also the C1E option to disable on AMD processor.


----- Mail original ----- 

De: "Somnath Roy" <Somnath.Roy@sandisk.com> 
À: "Andreas Bluemle" <andreas.bluemle@itxperts.de>, ceph-devel@vger.kernel.org 
Envoyé: Mercredi 8 Octobre 2014 19:38:26 
Objet: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params 

Thanks Andres for sharing this. I will try those out. 
BTW, I am using Ubuntu 14.04 LTS and couldn't find any sysfs entry like 'cpufreq'.. 

root@stormeap-4:~# ll /sys/devices/system/cpu/cpu10/ 
cache/ crash_notes driver/ microcode/ online subsystem/ topology/ 
cpuidle/ crash_notes_size firmware_node/ node0/ power/ thermal_throttle/ uevent 

I am using Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz. 

Regards 
Somnath 

-----Original Message----- 
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Andreas Bluemle 
Sent: Wednesday, October 08, 2014 9:33 AM 
To: ceph-devel@vger.kernel.org 
Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params 

Hi, 

as mentioned during today's meeting, here are the kernel boot parameters which I found to provide the basis for good performance results: 

processor.max_cstate=0 
intel_idle.max_cstate=0 

I understand these to basically turn off any power saving modes of the CPU; the CPU's we are using are like 
Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz 
Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz 

At the BIOS level, we 
- turn off Hyperthraeding 
- turn off Turbo mode (in order ot not leave the specifications) 
- turn on frequency floor override 

We also assert that 
/sys/devices/system/cpu/cpu*/cpufreq/scaling_governor 
is set to "performance" 

Using above we see a constant frequency at the maximum level allowed by the CPU (except Turbo mode). 


Best Regards 

Andreas Bluemle 






On Wed, 8 Oct 2014 02:51:21 +0200 
Mark Nelson <mark.nelson@inktank.com> wrote: 

> Hi All, 
> 
> Just a remind that the weekly performance meeting is on Wednesdays at 
> 8AM PST. Same bat time, same bat channel! 
> 
> Etherpad URL: 
> http://pad.ceph.com/p/performance_weekly 
> 
> To join the Meeting: 
> https://bluejeans.com/268261044 
> 
> To join via Browser: 
> https://bluejeans.com/268261044/browser 
> 
> To join with Lync: 
> https://bluejeans.com/268261044/lync 
> 
> 
> To join via Room System: 
> Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID: 
> 268261044 
> 
> To join via Phone: 
> 1) Dial: 
> +1 408 740 7256 
> +1 888 240 2560(US Toll Free) 
> +1 408 317 9253(Alternate Number) 
> (see all numbers - http://bluejeans.com/numbers) 
> 2) Enter Conference ID: 268261044 
> 
> Mark 
> -- 
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at http://vger.kernel.org/majordomo-info.html 
> 
> 



-- 
Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de 
ITXperts GmbH http://www.itxperts.de 
Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917 
D-81541 Muenchen (Germany) Fax: (+49) 89 89044910 

Company details: http://www.itxperts.de/imprint.htm 
-- 
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html 

________________________________ 

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). 

-- 
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
the body of a message to majordomo@vger.kernel.org 
More majordomo info at http://vger.kernel.org/majordomo-info.html 
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

[Stefan Priebe]

> Hi,
>
> as mentioned during today's meeting, here are the kernel boot parameters which I found to provide the basis for good performance results:
>
>    processor.max_cstate=0
>    intel_idle.max_cstate=0
>
> I understand these to basically turn off any power saving modes of the CPU; the CPU's we are using are like
>   Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
>   Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
>
> At the BIOS level, we
>   - turn off Hyperthraeding
>   - turn off Turbo mode (in order ot not leave the specifications)
>   - turn on frequency floor override
>
> We also assert that
>   /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
>   is set to "performance"
>
> Using above we see a constant frequency at the maximum level allowed by the CPU (except Turbo mode).

How much performance do we gain by this? Till now i thought it's just 
1-3% so i'm still running ondemand govenor plus power savings.

Greets,
Stefan

> Best Regards
>
> Andreas Bluemle
>
>
>
>
>
>
> On Wed, 8 Oct 2014 02:51:21 +0200
> Mark Nelson <mark.nelson@inktank.com> wrote:
>
>> Hi All,
>>
>> Just a remind that the weekly performance meeting is on Wednesdays at
>> 8AM PST.  Same bat time, same bat channel!
>>
>> Etherpad URL:
>> http://pad.ceph.com/p/performance_weekly
>>
>> To join the Meeting:
>> https://bluejeans.com/268261044
>>
>> To join via Browser:
>> https://bluejeans.com/268261044/browser
>>
>> To join with Lync:
>> https://bluejeans.com/268261044/lync
>>
>>
>> To join via Room System:
>> Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID:
>> 268261044
>>
>> To join via Phone:
>> 1) Dial:
>>             +1 408 740 7256
>>             +1 888 240 2560(US Toll Free)
>>             +1 408 317 9253(Alternate Number)
>>             (see all numbers - http://bluejeans.com/numbers)
>> 2) Enter Conference ID: 268261044
>>
>> Mark
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>> in the body of a message to majordomo@vger.kernel.org More majordomo
>> info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>
>
>
> --
> Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
> ITXperts GmbH                       http://www.itxperts.de
> Balanstrasse 73, Geb. 08            Phone: (+49) 89 89044917
> D-81541 Muenchen (Germany)          Fax:   (+49) 89 89044910
>
> Company details: http://www.itxperts.de/imprint.htm
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> ________________________________
>
> PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

[Paul Von-Stamwitz]

 
> > Hi,
> >
> > as mentioned during today's meeting, here are the kernel boot parameters
> which I found to provide the basis for good performance results:
> >
> >    processor.max_cstate=0
> >    intel_idle.max_cstate=0
> >
> > I understand these to basically turn off any power saving modes of the
> CPU; the CPU's we are using are like
> >   Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
> >   Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
> >
> > At the BIOS level, we
> >   - turn off Hyperthraeding
> >   - turn off Turbo mode (in order ot not leave the specifications)
> >   - turn on frequency floor override
> >
> > We also assert that
> >   /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
> >   is set to "performance"
> >
> > Using above we see a constant frequency at the maximum level allowed by
> the CPU (except Turbo mode).
> 
> How much performance do we gain by this? Till now i thought it's just 1-3% so
> i'm still running ondemand govenor plus power savings.

As always, it depends. I saw noticeable increases in some throughput tests (though I can't recall the % gain.) More important to me was that it made my fio results much more consistent. As we measure improvements, these settings remove some of the "system noise".

Best,
Paul

> 
> Greets,
> Stefan
> 
> > Best Regards
> >
> > Andreas Bluemle
> >
> >
> >
> >
> >
> >
> > On Wed, 8 Oct 2014 02:51:21 +0200
> > Mark Nelson <mark.nelson@inktank.com> wrote:
> >
> >> Hi All,
> >>
> >> Just a remind that the weekly performance meeting is on Wednesdays at
> >> 8AM PST.  Same bat time, same bat channel!
> >>
> >> Etherpad URL:
> >> http://pad.ceph.com/p/performance_weekly
> >>
> >> To join the Meeting:
> >> https://bluejeans.com/268261044
> >>
> >> To join via Browser:
> >> https://bluejeans.com/268261044/browser
> >>
> >> To join with Lync:
> >> https://bluejeans.com/268261044/lync
> >>
> >>
> >> To join via Room System:
> >> Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID:
> >> 268261044
> >>
> >> To join via Phone:
> >> 1) Dial:
> >>             +1 408 740 7256
> >>             +1 888 240 2560(US Toll Free)
> >>             +1 408 317 9253(Alternate Number)
> >>             (see all numbers - http://bluejeans.com/numbers)
> >> 2) Enter Conference ID: 268261044
> >>
> >> Mark
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> >> in the body of a message to majordomo@vger.kernel.org More
> majordomo
> >> info at  http://vger.kernel.org/majordomo-info.html
> >>
> >>
> >
> >
> >
> > --
> > Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
> > ITXperts GmbH                       http://www.itxperts.de
> > Balanstrasse 73, Geb. 08            Phone: (+49) 89 89044917
> > D-81541 Muenchen (Germany)          Fax:   (+49) 89 89044910
> >
> > Company details: http://www.itxperts.de/imprint.htm
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > in the body of a message to majordomo@vger.kernel.org More
> majordomo
> > info at  http://vger.kernel.org/majordomo-info.html
> >
> > ________________________________
> >
> > PLEASE NOTE: The information contained in this electronic mail message is
> intended only for the use of the designated recipient(s) named above. If the
> reader of this message is not the intended recipient, you are hereby notified
> that you have received this message in error and that any review,
> dissemination, distribution, or copying of this message is strictly prohibited. If
> you have received this communication in error, please notify the sender by
> telephone or e-mail (as shown above) immediately and destroy any and all
> copies of this message in your possession (whether hard copies or
> electronically stored copies).
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > in the body of a message to majordomo@vger.kernel.org More
> majordomo
> > info at  http://vger.kernel.org/majordomo-info.html
> >
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the
> body of a message to majordomo@vger.kernel.org More majordomo info at
> http://vger.kernel.org/majordomo-info.html

Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

[Andreas Bluemle]

[-- Attachment #1: Type: text/plain, Size: 5412 bytes --]

Hi,


On Wed, 8 Oct 2014 16:55:38 -0700
Paul Von-Stamwitz <PVonStamwitz@us.fujitsu.com> wrote:

>  
> > > Hi,
> > >
> > > as mentioned during today's meeting, here are the kernel boot
> > > parameters
> > which I found to provide the basis for good performance results:
> > >
> > >    processor.max_cstate=0
> > >    intel_idle.max_cstate=0
> > >
> > > I understand these to basically turn off any power saving modes
> > > of the
> > CPU; the CPU's we are using are like
> > >   Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
> > >   Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
> > >
> > > At the BIOS level, we
> > >   - turn off Hyperthraeding
> > >   - turn off Turbo mode (in order ot not leave the specifications)
> > >   - turn on frequency floor override
> > >
> > > We also assert that
> > >   /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
> > >   is set to "performance"
> > >
> > > Using above we see a constant frequency at the maximum level
> > > allowed by
> > the CPU (except Turbo mode).
> > 
> > How much performance do we gain by this? Till now i thought it's
> > just 1-3% so i'm still running ondemand govenor plus power savings.
> 
> As always, it depends. I saw noticeable increases in some throughput
> tests (though I can't recall the % gain.) More important to me was
> that it made my fio results much more consistent. As we measure
> improvements, these settings remove some of the "system noise".
> 
> Best,
> Paul
> 

There were two different aspects which showed improvemnt:
 - code was executed faster
 - thread switching delays were reduced significantly

See the attached grahics. They show processing of a 4 kB write
request: processing at the Pipe::Reader is roughly 200 us in both
pictures, and sth. like 20 us at the OSD::Dispatcher. So there
is not much of a benefit here.

But the delay between the end of the Pipe::Reader and the start
of the OSD::Dispatcher threads reduced really significantly.


(And sorry for the late response)


> > 
> > Greets,
> > Stefan
> > 
> > > Best Regards
> > >
> > > Andreas Bluemle
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Wed, 8 Oct 2014 02:51:21 +0200
> > > Mark Nelson <mark.nelson@inktank.com> wrote:
> > >
> > >> Hi All,
> > >>
> > >> Just a remind that the weekly performance meeting is on
> > >> Wednesdays at 8AM PST.  Same bat time, same bat channel!
> > >>
> > >> Etherpad URL:
> > >> http://pad.ceph.com/p/performance_weekly
> > >>
> > >> To join the Meeting:
> > >> https://bluejeans.com/268261044
> > >>
> > >> To join via Browser:
> > >> https://bluejeans.com/268261044/browser
> > >>
> > >> To join with Lync:
> > >> https://bluejeans.com/268261044/lync
> > >>
> > >>
> > >> To join via Room System:
> > >> Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID:
> > >> 268261044
> > >>
> > >> To join via Phone:
> > >> 1) Dial:
> > >>             +1 408 740 7256
> > >>             +1 888 240 2560(US Toll Free)
> > >>             +1 408 317 9253(Alternate Number)
> > >>             (see all numbers - http://bluejeans.com/numbers)
> > >> 2) Enter Conference ID: 268261044
> > >>
> > >> Mark
> > >> --
> > >> To unsubscribe from this list: send the line "unsubscribe
> > >> ceph-devel" in the body of a message to
> > >> majordomo@vger.kernel.org More
> > majordomo
> > >> info at  http://vger.kernel.org/majordomo-info.html
> > >>
> > >>
> > >
> > >
> > >
> > > --
> > > Andreas Bluemle
> > > mailto:Andreas.Bluemle@itxperts.de ITXperts
> > > GmbH                       http://www.itxperts.de Balanstrasse
> > > 73, Geb. 08            Phone: (+49) 89 89044917 D-81541 Muenchen
> > > (Germany)          Fax:   (+49) 89 89044910
> > >
> > > Company details: http://www.itxperts.de/imprint.htm
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe
> > > ceph-devel" in the body of a message to majordomo@vger.kernel.org
> > > More
> > majordomo
> > > info at  http://vger.kernel.org/majordomo-info.html
> > >
> > > ________________________________
> > >
> > > PLEASE NOTE: The information contained in this electronic mail
> > > message is
> > intended only for the use of the designated recipient(s) named
> > above. If the reader of this message is not the intended recipient,
> > you are hereby notified that you have received this message in
> > error and that any review, dissemination, distribution, or copying
> > of this message is strictly prohibited. If you have received this
> > communication in error, please notify the sender by telephone or
> > e-mail (as shown above) immediately and destroy any and all copies
> > of this message in your possession (whether hard copies or
> > electronically stored copies).
> > >
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe
> > > ceph-devel" in the body of a message to majordomo@vger.kernel.org
> > > More
> > majordomo
> > > info at  http://vger.kernel.org/majordomo-info.html
> > >
> > --
> > To unsubscribe from this list: send the line "unsubscribe
> > ceph-devel" in the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> 
> 



-- 
Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
ITXperts GmbH                       http://www.itxperts.de
Balanstrasse 73, Geb. 08            Phone: (+49) 89 89044917
D-81541 Muenchen (Germany)          Fax:   (+49) 89 89044910

Company details: http://www.itxperts.de/imprint.htm

[-- Attachment #2: detail-latency-default.png --]
[-- Type: image/png, Size: 12330 bytes --]

[-- Attachment #3: detail-latency-no-powersave-modes.png --]
[-- Type: image/png, Size: 11065 bytes --]

Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

[Sage Weil]

On Tue, 14 Oct 2014, Andreas Bluemle wrote:
> Hi,
> 
> 
> On Wed, 8 Oct 2014 16:55:38 -0700
> Paul Von-Stamwitz <PVonStamwitz@us.fujitsu.com> wrote:
> 
> >  
> > > > Hi,
> > > >
> > > > as mentioned during today's meeting, here are the kernel boot
> > > > parameters
> > > which I found to provide the basis for good performance results:
> > > >
> > > >    processor.max_cstate=0
> > > >    intel_idle.max_cstate=0
> > > >
> > > > I understand these to basically turn off any power saving modes
> > > > of the
> > > CPU; the CPU's we are using are like
> > > >   Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
> > > >   Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
> > > >
> > > > At the BIOS level, we
> > > >   - turn off Hyperthraeding
> > > >   - turn off Turbo mode (in order ot not leave the specifications)
> > > >   - turn on frequency floor override
> > > >
> > > > We also assert that
> > > >   /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
> > > >   is set to "performance"
> > > >
> > > > Using above we see a constant frequency at the maximum level
> > > > allowed by
> > > the CPU (except Turbo mode).
> > > 
> > > How much performance do we gain by this? Till now i thought it's
> > > just 1-3% so i'm still running ondemand govenor plus power savings.
> > 
> > As always, it depends. I saw noticeable increases in some throughput
> > tests (though I can't recall the % gain.) More important to me was
> > that it made my fio results much more consistent. As we measure
> > improvements, these settings remove some of the "system noise".
> > 
> > Best,
> > Paul
> > 
> 
> There were two different aspects which showed improvemnt:
>  - code was executed faster
>  - thread switching delays were reduced significantly
> 
> See the attached grahics. They show processing of a 4 kB write
> request: processing at the Pipe::Reader is roughly 200 us in both
> pictures, and sth. like 20 us at the OSD::Dispatcher. So there
> is not much of a benefit here.
> 
> But the delay between the end of the Pipe::Reader and the start
> of the OSD::Dispatcher threads reduced really significantly.

This test had a single outstanding IO, right?  The question for me is if 
this reflect latencies we'd see under a realistic workload, where the are 
more IOs in flight and the CPUs aren't likely to be in low power states.  
I'm not sure how low the load needs to be before those states kick in and 
these latencies start to appear...

sage

Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

[Andreas Bluemle]

Hi Sage,

[embedded below]

On Tue, 14 Oct 2014 06:13:58 -0700 (PDT)
Sage Weil <sage@newdream.net> wrote:

> On Tue, 14 Oct 2014, Andreas Bluemle wrote:
> > Hi,
> > 
> > 
> > On Wed, 8 Oct 2014 16:55:38 -0700
> > Paul Von-Stamwitz <PVonStamwitz@us.fujitsu.com> wrote:
> > 
> > >  
> > > > > Hi,
> > > > >
> > > > > as mentioned during today's meeting, here are the kernel boot
> > > > > parameters
> > > > which I found to provide the basis for good performance results:
> > > > >
> > > > >    processor.max_cstate=0
> > > > >    intel_idle.max_cstate=0
> > > > >
> > > > > I understand these to basically turn off any power saving
> > > > > modes of the
> > > > CPU; the CPU's we are using are like
> > > > >   Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
> > > > >   Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
> > > > >
> > > > > At the BIOS level, we
> > > > >   - turn off Hyperthraeding
> > > > >   - turn off Turbo mode (in order ot not leave the
> > > > > specifications)
> > > > >   - turn on frequency floor override
> > > > >
> > > > > We also assert that
> > > > >   /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
> > > > >   is set to "performance"
> > > > >
> > > > > Using above we see a constant frequency at the maximum level
> > > > > allowed by
> > > > the CPU (except Turbo mode).
> > > > 
> > > > How much performance do we gain by this? Till now i thought it's
> > > > just 1-3% so i'm still running ondemand govenor plus power
> > > > savings.
> > > 
> > > As always, it depends. I saw noticeable increases in some
> > > throughput tests (though I can't recall the % gain.) More
> > > important to me was that it made my fio results much more
> > > consistent. As we measure improvements, these settings remove
> > > some of the "system noise".
> > > 
> > > Best,
> > > Paul
> > > 
> > 
> > There were two different aspects which showed improvemnt:
> >  - code was executed faster
> >  - thread switching delays were reduced significantly
> > 
> > See the attached grahics. They show processing of a 4 kB write
> > request: processing at the Pipe::Reader is roughly 200 us in both
> > pictures, and sth. like 20 us at the OSD::Dispatcher. So there
> > is not much of a benefit here.
> > 
> > But the delay between the end of the Pipe::Reader and the start
> > of the OSD::Dispatcher threads reduced really significantly.
> 
> This test had a single outstanding IO, right?  The question for me is
> if this reflect latencies we'd see under a realistic workload, where
> the are more IOs in flight and the CPUs aren't likely to be in low
> power states. I'm not sure how low the load needs to be before those
> states kick in and these latencies start to appear...
> 
> sage

Yes and no...

Yes: the test was a fio sequential write, 4k per write, with a
single IO in flight.

No: this means that on a given object in the osd file store with the
default size of 4 MByte, 1024 subsequent write requests will hit that 
object - and hence the corresponding ceph-osd daemon. So even though
the system as a whole was not very busy, the ceph-osd daemon assigned
to the file object under pressure was fairly busy.

The intention of the test was to eliminate additional latencies
because of queues building up.

What the test shows is the contribution of the various processing
steps within ceph-osd to the overall latency for an individual
write requres when CPU power state related effects have been
eliminated,


> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 



-- 
Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
ITXperts GmbH                       http://www.itxperts.de
Balanstrasse 73, Geb. 08            Phone: (+49) 89 89044917
D-81541 Muenchen (Germany)          Fax:   (+49) 89 89044910

Company details: http://www.itxperts.de/imprint.htm

RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

[Sage Weil]

On Wed, 15 Oct 2014, Shu, Xinxin wrote:
> Hi all , recently we tested 4K random write performance on our full SSD 
> setup (12 x Intel DC3700) , but peak performance is ~23K IOPS, which is 
> much lower than hardware capability , with detail latency breakdown , we 
> found that most of latency comes from osd queue , we have noticed the 
> optimizations on osd queue , and tried latest master on our setup , but 
> there is a performance regression , we also checked the qlock and pg 
> lock with perf counter, the waiting count and latency are very small, 
> the attached pdf shows the details , any suggestion will be appreciated 
> ?

I would start by making sure 'osd enable op tracker = false' if it isn't 
already.

The other thing to keep in mind is that a lot of the work has enabled 
OSD perforamnce to scale as the clients increase.  It looks like 
your test has a single client.  Can you try running 2, 4, 8 clients 
and see if the per-OSD throughput goes up?

Digging into the code with a tool like vtune would be extremely helpful, I 
think.  There is a lot of time spent in do_op (osd prepare and osd queue) 
that fujitsu has called out but we haven't narrowed down where the time is 
being spent.

sage


> 
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Andreas Bluemle
> Sent: Tuesday, October 14, 2014 10:38 PM
> To: Sage Weil
> Cc: Paul Von-Stamwitz; Stefan Priebe; Somnath Roy; ceph-devel@vger.kernel.org
> Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
> 
> Hi Sage,
> 
> [embedded below]
> 
> On Tue, 14 Oct 2014 06:13:58 -0700 (PDT) Sage Weil <sage@newdream.net> wrote:
> 
> > On Tue, 14 Oct 2014, Andreas Bluemle wrote:
> > > Hi,
> > > 
> > > 
> > > On Wed, 8 Oct 2014 16:55:38 -0700
> > > Paul Von-Stamwitz <PVonStamwitz@us.fujitsu.com> wrote:
> > > 
> > > >  
> > > > > > Hi,
> > > > > >
> > > > > > as mentioned during today's meeting, here are the kernel boot 
> > > > > > parameters
> > > > > which I found to provide the basis for good performance results:
> > > > > >
> > > > > >    processor.max_cstate=0
> > > > > >    intel_idle.max_cstate=0
> > > > > >
> > > > > > I understand these to basically turn off any power saving 
> > > > > > modes of the
> > > > > CPU; the CPU's we are using are like
> > > > > >   Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
> > > > > >   Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
> > > > > >
> > > > > > At the BIOS level, we
> > > > > >   - turn off Hyperthraeding
> > > > > >   - turn off Turbo mode (in order ot not leave the
> > > > > > specifications)
> > > > > >   - turn on frequency floor override
> > > > > >
> > > > > > We also assert that
> > > > > >   /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
> > > > > >   is set to "performance"
> > > > > >
> > > > > > Using above we see a constant frequency at the maximum level 
> > > > > > allowed by
> > > > > the CPU (except Turbo mode).
> > > > > 
> > > > > How much performance do we gain by this? Till now i thought it's 
> > > > > just 1-3% so i'm still running ondemand govenor plus power 
> > > > > savings.
> > > > 
> > > > As always, it depends. I saw noticeable increases in some 
> > > > throughput tests (though I can't recall the % gain.) More 
> > > > important to me was that it made my fio results much more 
> > > > consistent. As we measure improvements, these settings remove some 
> > > > of the "system noise".
> > > > 
> > > > Best,
> > > > Paul
> > > > 
> > > 
> > > There were two different aspects which showed improvemnt:
> > >  - code was executed faster
> > >  - thread switching delays were reduced significantly
> > > 
> > > See the attached grahics. They show processing of a 4 kB write
> > > request: processing at the Pipe::Reader is roughly 200 us in both 
> > > pictures, and sth. like 20 us at the OSD::Dispatcher. So there is 
> > > not much of a benefit here.
> > > 
> > > But the delay between the end of the Pipe::Reader and the start of 
> > > the OSD::Dispatcher threads reduced really significantly.
> > 
> > This test had a single outstanding IO, right?  The question for me is 
> > if this reflect latencies we'd see under a realistic workload, where 
> > the are more IOs in flight and the CPUs aren't likely to be in low 
> > power states. I'm not sure how low the load needs to be before those 
> > states kick in and these latencies start to appear...
> > 
> > sage
> 
> Yes and no...
> 
> Yes: the test was a fio sequential write, 4k per write, with a single IO in flight.
> 
> No: this means that on a given object in the osd file store with the default size of 4 MByte, 1024 subsequent write requests will hit that object - and hence the corresponding ceph-osd daemon. So even though the system as a whole was not very busy, the ceph-osd daemon assigned to the file object under pressure was fairly busy.
> 
> The intention of the test was to eliminate additional latencies because of queues building up.
> 
> What the test shows is the contribution of the various processing steps within ceph-osd to the overall latency for an individual write requres when CPU power state related effects have been eliminated,
> 
> 
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > in the body of a message to majordomo@vger.kernel.org More majordomo 
> > info at  http://vger.kernel.org/majordomo-info.html
> > 
> > 
> 
> 
> 
> -- 
> Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
> ITXperts GmbH                       http://www.itxperts.de
> Balanstrasse 73, Geb. 08            Phone: (+49) 89 89044917
> D-81541 Muenchen (Germany)          Fax:   (+49) 89 89044910
> 
> Company details: http://www.itxperts.de/imprint.htm
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

[Somnath Roy]

Sage,
I think they seem to be using 7VM (and thus 7 librbd clients) clients for the test.
XinXin,
You are running 2 OSDS/SSD and that is not recommended . Not sure that has an impact or not.  Along with disabling optracker as Sage suggested, you may want to tweak the osd num shards and number of filestore threads to see if it is improving performance.
BTW, each librados client is now ~20% slower (even after rbd_cache = false) and with 7 clients adding those degradation could be significant. One quick check you can do to factor out librbd degradation, is to use firefly librbd/librados combination.

Thanks & Regards
Somnath

-----Original Message-----
From: Sage Weil [mailto:sage@newdream.net]
Sent: Tuesday, October 14, 2014 7:23 PM
To: Shu, Xinxin
Cc: Andreas Bluemle; Paul Von-Stamwitz; Stefan Priebe; Somnath Roy; ceph-devel@vger.kernel.org; Zhang, Jian
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

On Wed, 15 Oct 2014, Shu, Xinxin wrote:
> Hi all , recently we tested 4K random write performance on our full
> SSD setup (12 x Intel DC3700) , but peak performance is ~23K IOPS,
> which is much lower than hardware capability , with detail latency
> breakdown , we found that most of latency comes from osd queue , we
> have noticed the optimizations on osd queue , and tried latest master
> on our setup , but there is a performance regression , we also checked
> the qlock and pg lock with perf counter, the waiting count and latency
> are very small, the attached pdf shows the details , any suggestion
> will be appreciated ?

I would start by making sure 'osd enable op tracker = false' if it isn't already.

The other thing to keep in mind is that a lot of the work has enabled OSD perforamnce to scale as the clients increase.  It looks like your test has a single client.  Can you try running 2, 4, 8 clients and see if the per-OSD throughput goes up?

Digging into the code with a tool like vtune would be extremely helpful, I think.  There is a lot of time spent in do_op (osd prepare and osd queue) that fujitsu has called out but we haven't narrowed down where the time is being spent.

sage


>
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org
> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Andreas Bluemle
> Sent: Tuesday, October 14, 2014 10:38 PM
> To: Sage Weil
> Cc: Paul Von-Stamwitz; Stefan Priebe; Somnath Roy;
> ceph-devel@vger.kernel.org
> Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot
> params
>
> Hi Sage,
>
> [embedded below]
>
> On Tue, 14 Oct 2014 06:13:58 -0700 (PDT) Sage Weil <sage@newdream.net> wrote:
>
> > On Tue, 14 Oct 2014, Andreas Bluemle wrote:
> > > Hi,
> > >
> > >
> > > On Wed, 8 Oct 2014 16:55:38 -0700
> > > Paul Von-Stamwitz <PVonStamwitz@us.fujitsu.com> wrote:
> > >
> > > >
> > > > > > Hi,
> > > > > >
> > > > > > as mentioned during today's meeting, here are the kernel
> > > > > > boot parameters
> > > > > which I found to provide the basis for good performance results:
> > > > > >
> > > > > >    processor.max_cstate=0
> > > > > >    intel_idle.max_cstate=0
> > > > > >
> > > > > > I understand these to basically turn off any power saving
> > > > > > modes of the
> > > > > CPU; the CPU's we are using are like
> > > > > >   Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
> > > > > >   Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
> > > > > >
> > > > > > At the BIOS level, we
> > > > > >   - turn off Hyperthraeding
> > > > > >   - turn off Turbo mode (in order ot not leave the
> > > > > > specifications)
> > > > > >   - turn on frequency floor override
> > > > > >
> > > > > > We also assert that
> > > > > >   /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
> > > > > >   is set to "performance"
> > > > > >
> > > > > > Using above we see a constant frequency at the maximum level
> > > > > > allowed by
> > > > > the CPU (except Turbo mode).
> > > > >
> > > > > How much performance do we gain by this? Till now i thought
> > > > > it's just 1-3% so i'm still running ondemand govenor plus
> > > > > power savings.
> > > >
> > > > As always, it depends. I saw noticeable increases in some
> > > > throughput tests (though I can't recall the % gain.) More
> > > > important to me was that it made my fio results much more
> > > > consistent. As we measure improvements, these settings remove
> > > > some of the "system noise".
> > > >
> > > > Best,
> > > > Paul
> > > >
> > >
> > > There were two different aspects which showed improvemnt:
> > >  - code was executed faster
> > >  - thread switching delays were reduced significantly
> > >
> > > See the attached grahics. They show processing of a 4 kB write
> > > request: processing at the Pipe::Reader is roughly 200 us in both
> > > pictures, and sth. like 20 us at the OSD::Dispatcher. So there is
> > > not much of a benefit here.
> > >
> > > But the delay between the end of the Pipe::Reader and the start of
> > > the OSD::Dispatcher threads reduced really significantly.
> >
> > This test had a single outstanding IO, right?  The question for me
> > is if this reflect latencies we'd see under a realistic workload,
> > where the are more IOs in flight and the CPUs aren't likely to be in
> > low power states. I'm not sure how low the load needs to be before
> > those states kick in and these latencies start to appear...
> >
> > sage
>
> Yes and no...
>
> Yes: the test was a fio sequential write, 4k per write, with a single IO in flight.
>
> No: this means that on a given object in the osd file store with the default size of 4 MByte, 1024 subsequent write requests will hit that object - and hence the corresponding ceph-osd daemon. So even though the system as a whole was not very busy, the ceph-osd daemon assigned to the file object under pressure was fairly busy.
>
> The intention of the test was to eliminate additional latencies because of queues building up.
>
> What the test shows is the contribution of the various processing
> steps within ceph-osd to the overall latency for an individual write
> requres when CPU power state related effects have been eliminated,
>
>
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > in the body of a message to majordomo@vger.kernel.org More majordomo
> > info at  http://vger.kernel.org/majordomo-info.html
> >
> >
>
>
>
> --
> Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
> ITXperts GmbH                       http://www.itxperts.de
> Balanstrasse 73, Geb. 08            Phone: (+49) 89 89044917
> D-81541 Muenchen (Germany)          Fax:   (+49) 89 89044910
>
> Company details: http://www.itxperts.de/imprint.htm
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org More majordomo
> info at  http://vger.kernel.org/majordomo-info.html
>

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

[Shu, Xinxin]

Hi sage ,

With latest master , we do set 'osd_enable_op_tracker = false' , we tested up to 7 rbd clients in our test , but after two clients, the iops is stable at ~23K, there is no performance gain with more clients

-----Original Message-----
From: Sage Weil [mailto:sage@newdream.net] 
Sent: Wednesday, October 15, 2014 10:23 AM
To: Shu, Xinxin
Cc: Andreas Bluemle; Paul Von-Stamwitz; Stefan Priebe; Somnath Roy; ceph-devel@vger.kernel.org; Zhang, Jian
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params

On Wed, 15 Oct 2014, Shu, Xinxin wrote:
> Hi all , recently we tested 4K random write performance on our full 
> SSD setup (12 x Intel DC3700) , but peak performance is ~23K IOPS, 
> which is much lower than hardware capability , with detail latency 
> breakdown , we found that most of latency comes from osd queue , we 
> have noticed the optimizations on osd queue , and tried latest master 
> on our setup , but there is a performance regression , we also checked 
> the qlock and pg lock with perf counter, the waiting count and latency 
> are very small, the attached pdf shows the details , any suggestion 
> will be appreciated ?

I would start by making sure 'osd enable op tracker = false' if it isn't already.

The other thing to keep in mind is that a lot of the work has enabled OSD perforamnce to scale as the clients increase.  It looks like your test has a single client.  Can you try running 2, 4, 8 clients and see if the per-OSD throughput goes up?

Digging into the code with a tool like vtune would be extremely helpful, I think.  There is a lot of time spent in do_op (osd prepare and osd queue) that fujitsu has called out but we haven't narrowed down where the time is being spent.

sage


> 
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org 
> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Andreas Bluemle
> Sent: Tuesday, October 14, 2014 10:38 PM
> To: Sage Weil
> Cc: Paul Von-Stamwitz; Stefan Priebe; Somnath Roy; 
> ceph-devel@vger.kernel.org
> Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot 
> params
> 
> Hi Sage,
> 
> [embedded below]
> 
> On Tue, 14 Oct 2014 06:13:58 -0700 (PDT) Sage Weil <sage@newdream.net> wrote:
> 
> > On Tue, 14 Oct 2014, Andreas Bluemle wrote:
> > > Hi,
> > > 
> > > 
> > > On Wed, 8 Oct 2014 16:55:38 -0700
> > > Paul Von-Stamwitz <PVonStamwitz@us.fujitsu.com> wrote:
> > > 
> > > >  
> > > > > > Hi,
> > > > > >
> > > > > > as mentioned during today's meeting, here are the kernel 
> > > > > > boot parameters
> > > > > which I found to provide the basis for good performance results:
> > > > > >
> > > > > >    processor.max_cstate=0
> > > > > >    intel_idle.max_cstate=0
> > > > > >
> > > > > > I understand these to basically turn off any power saving 
> > > > > > modes of the
> > > > > CPU; the CPU's we are using are like
> > > > > >   Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
> > > > > >   Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
> > > > > >
> > > > > > At the BIOS level, we
> > > > > >   - turn off Hyperthraeding
> > > > > >   - turn off Turbo mode (in order ot not leave the
> > > > > > specifications)
> > > > > >   - turn on frequency floor override
> > > > > >
> > > > > > We also assert that
> > > > > >   /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
> > > > > >   is set to "performance"
> > > > > >
> > > > > > Using above we see a constant frequency at the maximum level 
> > > > > > allowed by
> > > > > the CPU (except Turbo mode).
> > > > > 
> > > > > How much performance do we gain by this? Till now i thought 
> > > > > it's just 1-3% so i'm still running ondemand govenor plus 
> > > > > power savings.
> > > > 
> > > > As always, it depends. I saw noticeable increases in some 
> > > > throughput tests (though I can't recall the % gain.) More 
> > > > important to me was that it made my fio results much more 
> > > > consistent. As we measure improvements, these settings remove 
> > > > some of the "system noise".
> > > > 
> > > > Best,
> > > > Paul
> > > > 
> > > 
> > > There were two different aspects which showed improvemnt:
> > >  - code was executed faster
> > >  - thread switching delays were reduced significantly
> > > 
> > > See the attached grahics. They show processing of a 4 kB write
> > > request: processing at the Pipe::Reader is roughly 200 us in both 
> > > pictures, and sth. like 20 us at the OSD::Dispatcher. So there is 
> > > not much of a benefit here.
> > > 
> > > But the delay between the end of the Pipe::Reader and the start of 
> > > the OSD::Dispatcher threads reduced really significantly.
> > 
> > This test had a single outstanding IO, right?  The question for me 
> > is if this reflect latencies we'd see under a realistic workload, 
> > where the are more IOs in flight and the CPUs aren't likely to be in 
> > low power states. I'm not sure how low the load needs to be before 
> > those states kick in and these latencies start to appear...
> > 
> > sage
> 
> Yes and no...
> 
> Yes: the test was a fio sequential write, 4k per write, with a single IO in flight.
> 
> No: this means that on a given object in the osd file store with the default size of 4 MByte, 1024 subsequent write requests will hit that object - and hence the corresponding ceph-osd daemon. So even though the system as a whole was not very busy, the ceph-osd daemon assigned to the file object under pressure was fairly busy.
> 
> The intention of the test was to eliminate additional latencies because of queues building up.
> 
> What the test shows is the contribution of the various processing 
> steps within ceph-osd to the overall latency for an individual write 
> requres when CPU power state related effects have been eliminated,
> 
> 
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > in the body of a message to majordomo@vger.kernel.org More majordomo 
> > info at  http://vger.kernel.org/majordomo-info.html
> > 
> > 
> 
> 
> 
> -- 
> Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
> ITXperts GmbH                       http://www.itxperts.de
> Balanstrasse 73, Geb. 08            Phone: (+49) 89 89044917
> D-81541 Muenchen (Germany)          Fax:   (+49) 89 89044910
> 
> Company details: http://www.itxperts.de/imprint.htm
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html
>