osd: new pool flags: noscrub, nodeep-scrub

osd: new pool flags: noscrub, nodeep-scrub

[Mykola Golub]

Hi,

I would like to add new pool flags: noscrub and nodeep-scrub, to be
able to control scrubbing on per pool basis. In our case it could be
helpful in order to disable scrubbing on cache pools, which does not
work well right now, but I can imagine other scenarios where it could
be useful too.

Before I created a pull request, I would like to see if other people
consider this useful or may be have some other suggestions?

-- 
Mykola Golub

Re: osd: new pool flags: noscrub, nodeep-scrub

[Gregory Farnum]

On Fri, Sep 11, 2015 at 7:42 AM, Mykola Golub <mgolub@mirantis.com> wrote:
> Hi,
>
> I would like to add new pool flags: noscrub and nodeep-scrub, to be
> able to control scrubbing on per pool basis. In our case it could be
> helpful in order to disable scrubbing on cache pools, which does not
> work well right now, but I can imagine other scenarios where it could
> be useful too.

Can you talk more about this? It sounds to me like maybe you dislike
the performance impact of scrubbing, but it's fairly important in
terms of data integrity. I don't think we want to permanently disable
them. A corruption in the cache pool isn't any less important than in
the backing pool — it will eventually get flushed, and it's where all
the reads will be handled!
-Greg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: osd: new pool flags: noscrub, nodeep-scrub

[Mykola Golub]

On Fri, Sep 11, 2015 at 11:08:29AM +0100, Gregory Farnum wrote:
> On Fri, Sep 11, 2015 at 7:42 AM, Mykola Golub <mgolub@mirantis.com> wrote:
> > Hi,
> >
> > I would like to add new pool flags: noscrub and nodeep-scrub, to be
> > able to control scrubbing on per pool basis. In our case it could be
> > helpful in order to disable scrubbing on cache pools, which does not
> > work well right now, but I can imagine other scenarios where it could
> > be useful too.
> 
> Can you talk more about this? It sounds to me like maybe you dislike
> the performance impact of scrubbing, but it's fairly important in
> terms of data integrity. I don't think we want to permanently disable
> them. A corruption in the cache pool isn't any less important than in
> the backing pool — it will eventually get flushed, and it's where all
> the reads will be handled!

I was talking about this:

http://tracker.ceph.com/issues/8752

(false-negative on a caching pool). Although the best solution is
definitely to fix the bug, I am not sure it will be resolved soon (the
bug is open for a year). Still these false-negatives are annoying, as
they complicate monitoring for real inconsistent pgs. In this case I
might want to disable periodic scrub for caching pools, as a
workaround (I could do scrub for them manually though).

This might be not the best example where these flags could be helpful
(I just came to the idea when thinking about a workaround for that
problem, and this looked useful to me in general). We already have
'ceph osd set no[deep-]scrub', and users use it to temporary resolve
high I/O load. Being able to do this per pool looks useful too.

You might have pools of different importance for you, and disabling
scrub for some of them might be ok.

-- 
Mykola Golub
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: osd: new pool flags: noscrub, nodeep-scrub

[Sage Weil]

On Fri, 11 Sep 2015, Mykola Golub wrote:
> On Fri, Sep 11, 2015 at 11:08:29AM +0100, Gregory Farnum wrote:
> > On Fri, Sep 11, 2015 at 7:42 AM, Mykola Golub <mgolub@mirantis.com> wrote:
> > > Hi,
> > >
> > > I would like to add new pool flags: noscrub and nodeep-scrub, to be
> > > able to control scrubbing on per pool basis. In our case it could be
> > > helpful in order to disable scrubbing on cache pools, which does not
> > > work well right now, but I can imagine other scenarios where it could
> > > be useful too.
> > 
> > Can you talk more about this? It sounds to me like maybe you dislike
> > the performance impact of scrubbing, but it's fairly important in
> > terms of data integrity. I don't think we want to permanently disable
> > them. A corruption in the cache pool isn't any less important than in
> > the backing pool ? it will eventually get flushed, and it's where all
> > the reads will be handled!
> 
> I was talking about this:
> 
> http://tracker.ceph.com/issues/8752
> 
> (false-negative on a caching pool). Although the best solution is
> definitely to fix the bug, I am not sure it will be resolved soon (the
> bug is open for a year). Still these false-negatives are annoying, as
> they complicate monitoring for real inconsistent pgs. In this case I
> might want to disable periodic scrub for caching pools, as a
> workaround (I could do scrub for them manually though).
> 
> This might be not the best example where these flags could be helpful
> (I just came to the idea when thinking about a workaround for that
> problem, and this looked useful to me in general). We already have
> 'ceph osd set no[deep-]scrub', and users use it to temporary resolve
> high I/O load. Being able to do this per pool looks useful too.
> 
> You might have pools of different importance for you, and disabling
> scrub for some of them might be ok.

I wonder if, in addition, we should also allow scrub and deep-scrub 
intervals to be set on a per-pool basis?

sage

Re: osd: new pool flags: noscrub, nodeep-scrub

[Sebastien Han]

[-- Attachment #1: Type: text/plain, Size: 2886 bytes --]

I like the idea of being able to control on which pool we want to run scrub on.
Some data might deserve better protection than others, it’s really up to the admin.

The default behaviour will be to apply scrub to default pool and every new created pools.
The operator can change this behaviour either with a config flag or directly with a ceph mon command.

> On 11 Sep 2015, at 14:59, Sage Weil <sweil@redhat.com> wrote:
> 
> On Fri, 11 Sep 2015, Mykola Golub wrote:
>> On Fri, Sep 11, 2015 at 11:08:29AM +0100, Gregory Farnum wrote:
>>> On Fri, Sep 11, 2015 at 7:42 AM, Mykola Golub <mgolub@mirantis.com> wrote:
>>>> Hi,
>>>> 
>>>> I would like to add new pool flags: noscrub and nodeep-scrub, to be
>>>> able to control scrubbing on per pool basis. In our case it could be
>>>> helpful in order to disable scrubbing on cache pools, which does not
>>>> work well right now, but I can imagine other scenarios where it could
>>>> be useful too.
>>> 
>>> Can you talk more about this? It sounds to me like maybe you dislike
>>> the performance impact of scrubbing, but it's fairly important in
>>> terms of data integrity. I don't think we want to permanently disable
>>> them. A corruption in the cache pool isn't any less important than in
>>> the backing pool ? it will eventually get flushed, and it's where all
>>> the reads will be handled!
>> 
>> I was talking about this:
>> 
>> http://tracker.ceph.com/issues/8752
>> 
>> (false-negative on a caching pool). Although the best solution is
>> definitely to fix the bug, I am not sure it will be resolved soon (the
>> bug is open for a year). Still these false-negatives are annoying, as
>> they complicate monitoring for real inconsistent pgs. In this case I
>> might want to disable periodic scrub for caching pools, as a
>> workaround (I could do scrub for them manually though).
>> 
>> This might be not the best example where these flags could be helpful
>> (I just came to the idea when thinking about a workaround for that
>> problem, and this looked useful to me in general). We already have
>> 'ceph osd set no[deep-]scrub', and users use it to temporary resolve
>> high I/O load. Being able to do this per pool looks useful too.
>> 
>> You might have pools of different importance for you, and disabling
>> scrub for some of them might be ok.
> 
> I wonder if, in addition, we should also allow scrub and deep-scrub
> intervals to be set on a per-pool basis?
> 
> sage
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


Cheers.
––––
Sébastien Han
Senior Cloud Architect

"Always give 100%. Unless you're giving blood."

Mail: seb@redhat.com
Address: 11 bis, rue Roquépine - 75008 Paris


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 842 bytes --]

Re: osd: new pool flags: noscrub, nodeep-scrub

[Mykola Golub]

On Fri, Sep 11, 2015 at 05:59:56AM -0700, Sage Weil wrote:

> I wonder if, in addition, we should also allow scrub and deep-scrub 
> intervals to be set on a per-pool basis?

ceph osd pool set <pool> [deep-]scrub_interval N ?

I could do this too.

-- 
Mykola Golub

Re: osd: new pool flags: noscrub, nodeep-scrub

[Wido den Hollander]

On 11-09-15 15:11, Sebastien Han wrote:
> I like the idea of being able to control on which pool we want to run scrub on.
> Some data might deserve better protection than others, it’s really up to the admin.
> 

Indeed. Let the admin be in control. If he/she doesn't want that pool to
be scrubbed Ceph shouldn't care.

Same like you can mount a filesystem with nobarrier if you want to. Not
that it is a wise thing to do.

> The default behaviour will be to apply scrub to default pool and every new created pools.
> The operator can change this behaviour either with a config flag or directly with a ceph mon command.
> 
>> On 11 Sep 2015, at 14:59, Sage Weil <sweil@redhat.com> wrote:
>>
>> On Fri, 11 Sep 2015, Mykola Golub wrote:
>>> On Fri, Sep 11, 2015 at 11:08:29AM +0100, Gregory Farnum wrote:
>>>> On Fri, Sep 11, 2015 at 7:42 AM, Mykola Golub <mgolub@mirantis.com> wrote:
>>>>> Hi,
>>>>>
>>>>> I would like to add new pool flags: noscrub and nodeep-scrub, to be
>>>>> able to control scrubbing on per pool basis. In our case it could be
>>>>> helpful in order to disable scrubbing on cache pools, which does not
>>>>> work well right now, but I can imagine other scenarios where it could
>>>>> be useful too.
>>>>
>>>> Can you talk more about this? It sounds to me like maybe you dislike
>>>> the performance impact of scrubbing, but it's fairly important in
>>>> terms of data integrity. I don't think we want to permanently disable
>>>> them. A corruption in the cache pool isn't any less important than in
>>>> the backing pool ? it will eventually get flushed, and it's where all
>>>> the reads will be handled!
>>>
>>> I was talking about this:
>>>
>>> http://tracker.ceph.com/issues/8752
>>>
>>> (false-negative on a caching pool). Although the best solution is
>>> definitely to fix the bug, I am not sure it will be resolved soon (the
>>> bug is open for a year). Still these false-negatives are annoying, as
>>> they complicate monitoring for real inconsistent pgs. In this case I
>>> might want to disable periodic scrub for caching pools, as a
>>> workaround (I could do scrub for them manually though).
>>>
>>> This might be not the best example where these flags could be helpful
>>> (I just came to the idea when thinking about a workaround for that
>>> problem, and this looked useful to me in general). We already have
>>> 'ceph osd set no[deep-]scrub', and users use it to temporary resolve
>>> high I/O load. Being able to do this per pool looks useful too.
>>>
>>> You might have pools of different importance for you, and disabling
>>> scrub for some of them might be ok.
>>
>> I wonder if, in addition, we should also allow scrub and deep-scrub
>> intervals to be set on a per-pool basis?
>>
>> sage
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
> Cheers.
> ––––
> Sébastien Han
> Senior Cloud Architect
> 
> "Always give 100%. Unless you're giving blood."
> 
> Mail: seb@redhat.com
> Address: 11 bis, rue Roquépine - 75008 Paris
> 
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: osd: new pool flags: noscrub, nodeep-scrub

[Andrey Korolyov]

On Fri, Sep 11, 2015 at 4:24 PM, Mykola Golub <mgolub@mirantis.com> wrote:
> On Fri, Sep 11, 2015 at 05:59:56AM -0700, Sage Weil wrote:
>
>> I wonder if, in addition, we should also allow scrub and deep-scrub
>> intervals to be set on a per-pool basis?
>
> ceph osd pool set <pool> [deep-]scrub_interval N ?

BTW it would be absolutely lovely to see a copy-aware scrubs, e.g.
parallel (deep-) scrubs on a non-intersecting set of PGs. Currently as
far as I can see if the scrub starts, the max_scrubs is in effect only
for a primary OSD, allowing situations when two scrubs, primary and
non-primary, can land on a same OSD.

Re: osd: new pool flags: noscrub, nodeep-scrub

[Mykola Golub]

On Fri, Sep 11, 2015 at 04:24:23PM +0300, Mykola Golub wrote:
> On Fri, Sep 11, 2015 at 05:59:56AM -0700, Sage Weil wrote:
> 
> > I wonder if, in addition, we should also allow scrub and deep-scrub 
> > intervals to be set on a per-pool basis?
> 
> ceph osd pool set <pool> [deep-]scrub_interval N ?
> 
> I could do this too.

https://github.com/ceph/ceph/pull/5922

-- 
Mykola Golub