From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Stillwell, Bryan" Subject: Re: Discuss: New default recovery config settings Date: Fri, 29 May 2015 18:56:33 -0400 Message-ID: References: <1939533999.9756941.1432935747903.JavaMail.zimbra@redhat.com> <1394947829.9758745.1432936033017.JavaMail.zimbra@redhat.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0687108916==" Return-path: In-Reply-To: Content-Language: en-US List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ceph-users-bounces-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org Sender: "ceph-users" To: Josef Johansson , Samuel Just , ceph-devel , "'ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org' (ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org)" List-Id: ceph-devel.vger.kernel.org --===============0687108916== Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_D18E470A6229bryanstillwelltwcablecom_" --_000_D18E470A6229bryanstillwelltwcablecom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable I like the idea of turning the defaults down. During the ceph operators se= ssion at the OpenStack conference last week Warren described the behavior p= retty accurately as "Ceph basically DOSes itself unless you reduce those se= ttings." Maybe this is more of a problem when the clusters are small? Another idea would be to have a better way to prioritize recovery traffic t= o an even lower priority level by setting the ionice value to 'Idle' in the= CFQ scheduler? Bryan From: Josef Johansson > Date: Friday, May 29, 2015 at 4:16 PM To: Samuel Just >, ceph-devel >, "'ceph-users@= lists.ceph.com' (ceph-users-idqoXFIVOFKIjjVqG0RrOw@public.gmane.org= om)" > Subject: Re: [ceph-users] Discuss: New default recovery config settings Hi, We did it the other way around instead, defining a period where the load is= lighter and turn off/on backfill/recover. Then you want the backfill value= s to be the what is default right now. Also, someone said that (think it was Greg?) If you have problems with back= fill, your cluster backing store is not fast enough/too much load. If 10 osds goes down at the same time you want those values to be high to m= inimize the downtime. /Josef fre 29 maj 2015 23:47 Samuel Just > skrev: Many people have reported that they need to lower the osd recovery config o= ptions to minimize the impact of recovery on client io. We are talking abo= ut changing the defaults as follows: osd_max_backfills to 1 (from 10) osd_recovery_max_active to 3 (from 15) osd_recovery_op_priority to 1 (from 10) osd_recovery_max_single_start to 1 (from 5) We'd like a bit of feedback first though. Is anyone happy with the current= configs? Is anyone using something between these values and the current d= efaults? What kind of workload? I'd guess that lowering osd_max_backfills= to 1 is probably a good idea, but I wonder whether lowering osd_recovery_m= ax_active and osd_recovery_max_single_start will cause small objects to rec= over unacceptably slowly. Thoughts? -Sam _______________________________________________ ceph-users mailing list ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ________________________________ This E-mail and any of its attachments may contain Time Warner Cable propri= etary information, which is privileged, confidential, or subject to copyrig= ht belonging to Time Warner Cable. This E-mail is intended solely for the u= se of the individual or entity to which it is addressed. If you are not the= intended recipient of this E-mail, you are hereby notified that any dissem= ination, distribution, copying, or action taken in relation to the contents= of and attachments to this E-mail is strictly prohibited and may be unlawf= ul. If you have received this E-mail in error, please notify the sender imm= ediately and permanently delete the original and any copy of this E-mail an= d any printout. --_000_D18E470A6229bryanstillwelltwcablecom_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
I like the idea of turning the defaults down.  During the ceph op= erators session at the OpenStack conference last week Warren described the = behavior pretty accurately as "Ceph basically DOSes itself unless you = reduce those settings."  Maybe this is more of a problem when the clusters are small?

Another idea would be to have a better way to prioritize recovery traf= fic to an even lower priority level by setting the ionice value to 'Idle' i= n the CFQ scheduler?

Bryan



Hi,

We did it the other way around instead, defining a period wh= ere the load is lighter and turn off/on backfill/recover. Then you want the= backfill values to be the what is default right now.

Also, someone said that (think it was Greg?) If you have pro= blems with backfill, your cluster backing store is not fast enough/too much= load.
If 10 osds goes down at the same time you want those values to be high to m= inimize the downtime.

/Josef


fre 29 maj 2015 23:47 Samuel Just <sjust-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> skrev:
Many people have reported that they need to lower the osd recovery config o= ptions to minimize the impact of recovery on client io.  We are talkin= g about changing the defaults as follows:

osd_max_backfills to 1 (from 10)
osd_recovery_max_active to 3 (from 15)
osd_recovery_op_priority to 1 (from 10)
osd_recovery_max_single_start to 1 (from 5)

We'd like a bit of feedback first though.  Is anyone happy with the cu= rrent configs?  Is anyone using something between these values and the= current defaults?  What kind of workload?  I'd guess that loweri= ng osd_max_backfills to 1 is probably a good idea, but I wonder whether lowering osd_recovery_max_active and osd_recovery_max_sin= gle_start will cause small objects to recover unacceptably slowly.

Thoughts?
-Sam
_______________________________________________
ceph-users mailing list
ceph-users@l= ists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

This E-mail and any of its a= ttachments may contain Time Warner Cable proprietary information, which is = privileged, confidential, or subject to copyright belonging to Time Warner = Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you a= re not the intended recipient of this E-mail, you are hereby notified that = any dissemination, distribution, copying, or action taken in relation to th= e contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have receiv= ed this E-mail in error, please notify the sender immediately and permanent= ly delete the original and any copy of this E-mail and any printout.
 --_000_D18E470A6229bryanstillwelltwcablecom_-- --===============0687108916== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ ceph-users mailing list ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com --===============0687108916==--