maintanance on osd host

maintanance on osd host

[Stefan Priebe - Profihost AG]

Hi list,

how can i do a short maintanance like a kernel upgrade on an osd host?
Right now ceph starts to backfill immediatly if i say:
ceph osd out 41
...

Without ceph osd out command all clients hang for the time ceph does not
know that the host was rebootet.

I tried
ceph osd set nodown and ceph osd set noout
but this doesn't result in any difference

Stefan

Re: maintanance on osd host

[Andrey Korolyov]

On Tue, Feb 26, 2013 at 6:56 PM, Stefan Priebe - Profihost AG
<s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org> wrote:
> Hi list,
>
> how can i do a short maintanance like a kernel upgrade on an osd host?
> Right now ceph starts to backfill immediatly if i say:
> ceph osd out 41
> ...
>
> Without ceph osd out command all clients hang for the time ceph does not
> know that the host was rebootet.
>
> I tried
> ceph osd set nodown and ceph osd set noout
> but this doesn't result in any difference
>

Hi Stefan,

in my practice nodown will freeze all I/O for sure until OSD will
return, killing osd process and setting ``mon osd down out interval''
large enough will do the trick - you`ll get only two small freezes on
the peering process at start and at the end. Also it is very strange
that your clients hanging for a long time - I have set non-optimal
values for purpose and was not able to observe re-peering process
longer than a minute.

Re: maintanance on osd host

[Sage Weil]

On Tue, 26 Feb 2013, Stefan Priebe - Profihost AG wrote:
> Hi list,
> 
> how can i do a short maintanance like a kernel upgrade on an osd host?
> Right now ceph starts to backfill immediatly if i say:
> ceph osd out 41
> ...
> 
> Without ceph osd out command all clients hang for the time ceph does not
> know that the host was rebootet.
> 
> I tried
> ceph osd set nodown and ceph osd set noout
> but this doesn't result in any difference

For a temporary event like this, you want the osd to be down (so that io 
can continue with remaining replicas) but NOT to mark it out (so that data 
doesn't get rebalanced).  The simplest way to do that is

 ceph osd set noout
 killall ceph-osd
 .. reboot ..

Just remember to do

 ceph osd unset noout

when you are done so that future osds that fail will get marked out on 
their own after the 5 minute (default) interval.

sage

Re: maintanance on osd host

[Stefan Priebe - Profihost AG]

But that redults in a 1-3s hickup for all KVM vms. This is not what I want.

Stefan

Am 26.02.2013 um 18:06 schrieb Sage Weil <sage@inktank.com>:

> On Tue, 26 Feb 2013, Stefan Priebe - Profihost AG wrote:
>> Hi list,
>> 
>> how can i do a short maintanance like a kernel upgrade on an osd host?
>> Right now ceph starts to backfill immediatly if i say:
>> ceph osd out 41
>> ...
>> 
>> Without ceph osd out command all clients hang for the time ceph does not
>> know that the host was rebootet.
>> 
>> I tried
>> ceph osd set nodown and ceph osd set noout
>> but this doesn't result in any difference
> 
> For a temporary event like this, you want the osd to be down (so that io 
> can continue with remaining replicas) but NOT to mark it out (so that data 
> doesn't get rebalanced).  The simplest way to do that is
> 
> ceph osd set noout
> killall ceph-osd
> .. reboot ..
> 
> Just remember to do
> 
> ceph osd unset noout
> 
> when you are done so that future osds that fail will get marked out on 
> their own after the 5 minute (default) interval.
> 
> sage
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: maintanance on osd host

[Sage Weil]

On Tue, 26 Feb 2013, Stefan Priebe - Profihost AG wrote:
> But that redults in a 1-3s hickup for all KVM vms. This is not what I want.

You can do

 kill $pid
 ceph osd down $osdid

(or even reverse the order, if the sequence is quick enough) to avoid 
waiting for the failure detection delay.  But if the OSDs are going down, 
then the peering has to happen one way or another.

sage


> 
> Stefan
> 
> Am 26.02.2013 um 18:06 schrieb Sage Weil <sage@inktank.com>:
> 
> > On Tue, 26 Feb 2013, Stefan Priebe - Profihost AG wrote:
> >> Hi list,
> >> 
> >> how can i do a short maintanance like a kernel upgrade on an osd host?
> >> Right now ceph starts to backfill immediatly if i say:
> >> ceph osd out 41
> >> ...
> >> 
> >> Without ceph osd out command all clients hang for the time ceph does not
> >> know that the host was rebootet.
> >> 
> >> I tried
> >> ceph osd set nodown and ceph osd set noout
> >> but this doesn't result in any difference
> > 
> > For a temporary event like this, you want the osd to be down (so that io 
> > can continue with remaining replicas) but NOT to mark it out (so that data 
> > doesn't get rebalanced).  The simplest way to do that is
> > 
> > ceph osd set noout
> > killall ceph-osd
> > .. reboot ..
> > 
> > Just remember to do
> > 
> > ceph osd unset noout
> > 
> > when you are done so that future osds that fail will get marked out on 
> > their own after the 5 minute (default) interval.
> > 
> > sage
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: maintanance on osd host

[Stefan Priebe]

Hi Sage,

Am 26.02.2013 18:24, schrieb Sage Weil:
> On Tue, 26 Feb 2013, Stefan Priebe - Profihost AG wrote:
>> But that redults in a 1-3s hickup for all KVM vms. This is not what I want.
>
> You can do
>
>   kill $pid
>   ceph osd down $osdid
>
> (or even reverse the order, if the sequence is quick enough) to avoid
> waiting for the failure detection delay.  But if the OSDs are going down,
> then the peering has to happen one way or another.

But exaclty this results in starting backfill immediatly. My idea was to 
first mark the osd down so the mon knows about this fact. So no I/O is 
stalled. And then reboot the whole host but exactly this does not work 
like expected as backfilling is starting immediatly after setting the 
osd to down ;-(

Greets,
Stefan

Re: maintanance on osd host

[Sage Weil]

On Tue, 26 Feb 2013, Stefan Priebe wrote:
> Hi Sage,
> 
> Am 26.02.2013 18:24, schrieb Sage Weil:
> > On Tue, 26 Feb 2013, Stefan Priebe - Profihost AG wrote:
> > > But that redults in a 1-3s hickup for all KVM vms. This is not what I
> > > want.
> > 
> > You can do
> > 
> >   kill $pid
> >   ceph osd down $osdid
> > 
> > (or even reverse the order, if the sequence is quick enough) to avoid
> > waiting for the failure detection delay.  But if the OSDs are going down,
> > then the peering has to happen one way or another.
> 
> But exaclty this results in starting backfill immediatly. My idea was to first
> mark the osd down so the mon knows about this fact. So no I/O is stalled. And
> then reboot the whole host but exactly this does not work like expected as
> backfilling is starting immediatly after setting the osd to down ;-(

Backfilling should not happen on down, unless you have reconfigured 'mon 
osd down out interval = 0' or something along those lines.  Setting the 
'noout' flag will also prevent the osds from marking out.

As for limiting the IO stall: you could also do 'ceph osd set noup', 
then mark them down, then kill the daemon, and you won't have to worry 
about racing with the daemon marking itself back up (as it normally does).

sage

Re: maintanance on osd host

[Gregory Farnum]

On Tue, Feb 26, 2013 at 11:44 AM, Stefan Priebe <s.priebe@profihost.ag> wrote:
> Hi Sage,
>
> Am 26.02.2013 18:24, schrieb Sage Weil:
>
>> On Tue, 26 Feb 2013, Stefan Priebe - Profihost AG wrote:
>>>
>>> But that redults in a 1-3s hickup for all KVM vms. This is not what I
>>> want.
>>
>>
>> You can do
>>
>>   kill $pid
>>   ceph osd down $osdid
>>
>> (or even reverse the order, if the sequence is quick enough) to avoid
>> waiting for the failure detection delay.  But if the OSDs are going down,
>> then the peering has to happen one way or another.
>
>
> But exaclty this results in starting backfill immediatly. My idea was to
> first mark the osd down so the mon knows about this fact. So no I/O is
> stalled. And then reboot the whole host but exactly this does not work like
> expected as backfilling is starting immediatly after setting the osd to down
> ;-(

"out" and "down" are quite different — are you sure you tried "down"
and not "out"? (You reference out in your first email, rather than
down.)
-Greg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: maintanance on osd host

[Stefan Priebe - Profihost AG]

Hi Greg,
  Hi Sage,

Am 26.02.2013 21:27, schrieb Gregory Farnum:
> On Tue, Feb 26, 2013 at 11:44 AM, Stefan Priebe <s.priebe@profihost.ag> wrote:
> "out" and "down" are quite different — are you sure you tried "down"
> and not "out"? (You reference out in your first email, rather than
> down.)
> -Greg

sorry that's it i misread down / out. Sorry. Wouldn't it make sense to
mark the osd automatically down when shutting down via the init script?
It doesn't seem to make sense to hope for the automatic detection when
somebody uses the init script.

Stefan
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: maintanance on osd host

[Gregory Farnum]

On Tue, Feb 26, 2013 at 11:37 PM, Stefan Priebe - Profihost AG
<s.priebe@profihost.ag> wrote:
> Hi Greg,
>   Hi Sage,
>
> Am 26.02.2013 21:27, schrieb Gregory Farnum:
>> On Tue, Feb 26, 2013 at 11:44 AM, Stefan Priebe <s.priebe@profihost.ag> wrote:
>> "out" and "down" are quite different — are you sure you tried "down"
>> and not "out"? (You reference out in your first email, rather than
>> down.)
>> -Greg
>
> sorry that's it i misread down / out. Sorry. Wouldn't it make sense to
> mark the osd automatically down when shutting down via the init script?
> It doesn't seem to make sense to hope for the automatic detection when
> somebody uses the init script.

Yes, yes it would. http://tracker.ceph.com/issues/4267 :)
-Greg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: maintanance on osd host

[John Wilkins]

I have it documented here:

http://ceph.com/docs/master/rados/operations/troubleshooting-osd/#stopping-w-out-rebalancing

Let me know if this works for you.

On Thu, Feb 28, 2013 at 8:14 AM, Gregory Farnum <greg@inktank.com> wrote:
> On Tue, Feb 26, 2013 at 11:37 PM, Stefan Priebe - Profihost AG
> <s.priebe@profihost.ag> wrote:
>> Hi Greg,
>>   Hi Sage,
>>
>> Am 26.02.2013 21:27, schrieb Gregory Farnum:
>>> On Tue, Feb 26, 2013 at 11:44 AM, Stefan Priebe <s.priebe@profihost.ag> wrote:
>>> "out" and "down" are quite different — are you sure you tried "down"
>>> and not "out"? (You reference out in your first email, rather than
>>> down.)
>>> -Greg
>>
>> sorry that's it i misread down / out. Sorry. Wouldn't it make sense to
>> mark the osd automatically down when shutting down via the init script?
>> It doesn't seem to make sense to hope for the automatic detection when
>> somebody uses the init script.
>
> Yes, yes it would. http://tracker.ceph.com/issues/4267 :)
> -Greg
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
John Wilkins
Senior Technical Writer
Intank
john.wilkins@inktank.com
(415) 425-9599
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: maintanance on osd host

[Sylvain Munaut]

Hi,

> I have it documented here:
>
> http://ceph.com/docs/master/rados/operations/troubleshooting-osd/#stopping-w-out-rebalancing

That looks wrong to me

AFAIU it should be 'noout'. You want it marked down ASAP.

Cheers,

    Sylvain

Re: maintanance on osd host

[John Wilkins]

thanks. I've updated the docs accordingly. The change should be up in
a few minutes.

On Fri, Mar 1, 2013 at 12:38 PM, Sylvain Munaut
<s.munaut@whatever-company.com> wrote:
> Hi,
>
>> I have it documented here:
>>
>> http://ceph.com/docs/master/rados/operations/troubleshooting-osd/#stopping-w-out-rebalancing
>
> That looks wrong to me
>
> AFAIU it should be 'noout'. You want it marked down ASAP.
>
> Cheers,
>
>     Sylvain



-- 
John Wilkins
Senior Technical Writer
Intank
john.wilkins@inktank.com
(415) 425-9599