How to reduce the influenct on the IO when an osd is marked out?

How to reduce the influenct on the IO when an osd is marked out?

[wangsongbo]

Hi all,
when an osd is marked out, relative IO will be blocked, in which case, 
application built on ceph will fail.According to test result, the larger 
a data is,the longer it will take to elapse.
How to reduce the impact of this process on the IO?


Thanks and Regards,
WangSongbo

Re: [ceph-users] How to reduce the influenct on the IO when an osd is marked out?

[Martin Bureau]

Hello,

You can use these settings to lower the impact of backfilling and recovery operations:

osd max backfills
osd recovery max active

Additionally, if you are using the cfq io scheduler, with these you can set the io priority of recovery operations versus client operations:

osd client op priority
osd recovery op priority

Finally, if the osd is down temporarily because of a maintenance for example, you can set the noout flag (ceph osd set noout) so that backfilling will not occur during this time.

Regards,
Martin

________________________________________
From: ceph-users <ceph-users-bounces@lists.ceph.com> on behalf of wangsongbo <songbo1227@gmail.com>
Sent: October 9, 2015 11:52 PM
To: ceph-devel; ceph-users@ceph.com
Subject: [ceph-users] How to reduce the influenct on the IO when an osd is      marked out?

Hi all,
when an osd is marked out, relative IO will be blocked, in which case,
application built on ceph will fail.According to test result, the larger
a data is,the longer it will take to elapse.
How to reduce the impact of this process on the IO?


Thanks and Regards,
WangSongbo
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: How to reduce the influenct on the IO when an osd is marked out?

[Sage Weil]

[dropping ceph-users]

On Sat, 10 Oct 2015, wangsongbo wrote:
> Hi all,
> when an osd is marked out, relative IO will be blocked, in which case,
> application built on ceph will fail.According to test result, the larger a
> data is,the longer it will take to elapse.
> How to reduce the impact of this process on the IO?

When you mark an osd out the mon is doing prime_pg_temp, which 
preemptively remaps the PG to the same OSDs.  This should make peering 
fast... except that the OSDs still have to do a cycle of up_thru updates.

Sam, I think we can do the following to avoid this:

 - in build prior, we can infer that that interval for last_epoch_start is 
also a rw interval (because clearly it finished peering).

 - if the acting set and primary do not change, we can skip the up_thru 
update (because we will already infer rw from above).

I think the only caveat is that we can only skip the up_thru update 
once the entire cluster has a feature bit indicating they understand 
that last_epoch_started implies rw.

Anyway, this would mean that there's no subsequent mon interaction after 
the mark out (and probably lots of other common scenarios)...

What do you think?
sage