Best practice with 0.48.2 to take a node into maintenance

Best practice with 0.48.2 to take a node into maintenance

[Oliver Francke]

Hi *,

well, even if 0.48.2 is really stable and reliable, it is not everytime the case with linux kernel. We have a couple of nodes, where an update would make life better.
So, as our OSD-nodes have to care for VM's too, it's not the problem to let them drain so migrate all of them to other nodes.
Just reboot? Perhaps not, cause all OSD's will begin to remap/backfill, they are instructed to do so. Well, declare them as "osd lost"?
Dangerous. Is there another way I miss in doing node-maintenance? Will we have to wait for bobtail for far less hassle with all remapping and resources?

Thnx for comments,

Oliver.

Re: Best practice with 0.48.2 to take a node into maintenance

[Josh Durgin]

On 12/03/2012 11:05 AM, Oliver Francke wrote:
> Hi *,
>
> well, even if 0.48.2 is really stable and reliable, it is not everytime the case with linux kernel. We have a couple of nodes, where an update would make life better.
> So, as our OSD-nodes have to care for VM's too, it's not the problem to let them drain so migrate all of them to other nodes.
> Just reboot? Perhaps not, cause all OSD's will begin to remap/backfill, they are instructed to do so. Well, declare them as "osd lost"?
> Dangerous. Is there another way I miss in doing node-maintenance? Will we have to wait for bobtail for far less hassle with all remapping and resources?

By default the monitors won't mark an OSD out in the time it takes to
reboot, but if maintenance takes longer, you can drain data from the
node.

A simple way to rate limit it yourself is by slowly lowering the
weights of the OSDs on the host you want to update, e.g. by 0.1 at a
time and waiting for recovery to complete before lowering again. Once
they're at 0 and the cluster is healthy, they're not responsible for
any data anymore, and the node can be rebooted.

Josh

Re: Best practice with 0.48.2 to take a node into maintenance

[Oliver Francke]

Hi Josh,

Am 03.12.2012 um 20:14 schrieb Josh Durgin <josh.durgin@inktank.com>:

> On 12/03/2012 11:05 AM, Oliver Francke wrote:
>> Hi *,
>> 
>> well, even if 0.48.2 is really stable and reliable, it is not everytime the case with linux kernel. We have a couple of nodes, where an update would make life better.
>> So, as our OSD-nodes have to care for VM's too, it's not the problem to let them drain so migrate all of them to other nodes.
>> Just reboot? Perhaps not, cause all OSD's will begin to remap/backfill, they are instructed to do so. Well, declare them as "osd lost"?
>> Dangerous. Is there another way I miss in doing node-maintenance? Will we have to wait for bobtail for far less hassle with all remapping and resources?
> 
> By default the monitors won't mark an OSD out in the time it takes to
> reboot, but if maintenance takes longer, you can drain data from the
> node.
> 
> A simple way to rate limit it yourself is by slowly lowering the
> weights of the OSDs on the host you want to update, e.g. by 0.1 at a
> time and waiting for recovery to complete before lowering again. Once
> they're at 0 and the cluster is healthy, they're not responsible for
> any data anymore, and the node can be rebooted.
> 

true. Should have mentioned knowing smooth way. But for a planned reboot this take way too much time ;)
But if it's recommended, it's recommended ;)

Oliver.

> Josh
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Best practice with 0.48.2 to take a node into maintenance

[Smart Weblications GmbH - Florian Wiessner]

Am 03.12.2012 20:21, schrieb Oliver Francke:
> Hi Josh,
> 
> Am 03.12.2012 um 20:14 schrieb Josh Durgin <josh.durgin@inktank.com>:
> 
>> On 12/03/2012 11:05 AM, Oliver Francke wrote:
>>> Hi *,
>>>
>>> well, even if 0.48.2 is really stable and reliable, it is not everytime the case with linux kernel. We have a couple of nodes, where an update would make life better.
>>> So, as our OSD-nodes have to care for VM's too, it's not the problem to let them drain so migrate all of them to other nodes.
>>> Just reboot? Perhaps not, cause all OSD's will begin to remap/backfill, they are instructed to do so. Well, declare them as "osd lost"?
>>> Dangerous. Is there another way I miss in doing node-maintenance? Will we have to wait for bobtail for far less hassle with all remapping and resources?
>>
>> By default the monitors won't mark an OSD out in the time it takes to
>> reboot, but if maintenance takes longer, you can drain data from the
>> node.
>>
>> A simple way to rate limit it yourself is by slowly lowering the
>> weights of the OSDs on the host you want to update, e.g. by 0.1 at a
>> time and waiting for recovery to complete before lowering again. Once
>> they're at 0 and the cluster is healthy, they're not responsible for
>> any data anymore, and the node can be rebooted.
>>
> 
> true. Should have mentioned knowing smooth way. But for a planned reboot this take way too much time ;)
> But if it's recommended, it's recommended ;)
> 


I did rolling reboots of our whole cluster a few days ago (3.4.20). When the
system reboots and no fsck is done, ceph won't start to backfill in my setup.

I had some nodes do fsck after upgrade so ceph marked the osd as down and
started to backfill, but once the missing osd was back up running again, the
backfill stopped and ceph did just a little bit of peering and was healthy in a
few minutes again (2-5 minutes)...




-- 

Mit freundlichen Grüßen,

Florian Wiessner

Smart Weblications GmbH
Martinsberger Str. 1
D-95119 Naila

fon.: +49 9282 9638 200
fax.: +49 9282 9638 205
24/7: +49 900 144 000 00 - 0,99 EUR/Min*
http://www.smart-weblications.de

--
Sitz der Gesellschaft: Naila
Geschäftsführer: Florian Wiessner
HRB-Nr.: HRB 3840 Amtsgericht Hof
*aus dem dt. Festnetz, ggf. abweichende Preise aus dem Mobilfunknetz
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Best practice with 0.48.2 to take a node into maintenance

[Christopher Kunz]

Am 03.12.12 20:14, schrieb Josh Durgin:
> On 12/03/2012 11:05 AM, Oliver Francke wrote:
>> Hi *,
>>
>> well, even if 0.48.2 is really stable and reliable, it is not
>> everytime the case with linux kernel. We have a couple of nodes,
>> where an update would make life better.
>> So, as our OSD-nodes have to care for VM's too, it's not the problem
>> to let them drain so migrate all of them to other nodes.
>> Just reboot? Perhaps not, cause all OSD's will begin to
>> remap/backfill, they are instructed to do so. Well, declare them as
>> "osd lost"?
>> Dangerous. Is there another way I miss in doing node-maintenance?
>> Will we have to wait for bobtail for far less hassle with all
>> remapping and resources?
>
> By default the monitors won't mark an OSD out in the time it takes to
> reboot, but if maintenance takes longer, you can drain data from the
> node.
Hi,

what time is that (in seconds) and how can we reliably test this?

Regards,

--ck

Re: Best practice with 0.48.2 to take a node into maintenance

[Oliver Francke]

Hi Florian,

Am 03.12.2012 um 20:45 schrieb Smart Weblications GmbH - Florian Wiessner  <f.wiessner@smart-weblications.de>:

> Am 03.12.2012 20:21, schrieb Oliver Francke:
>> Hi Josh,
>> 
>> Am 03.12.2012 um 20:14 schrieb Josh Durgin <josh.durgin@inktank.com>:
>> 
>>> On 12/03/2012 11:05 AM, Oliver Francke wrote:
>>>> Hi *,
>>>> 
>>>> well, even if 0.48.2 is really stable and reliable, it is not everytime the case with linux kernel. We have a couple of nodes, where an update would make life better.
>>>> So, as our OSD-nodes have to care for VM's too, it's not the problem to let them drain so migrate all of them to other nodes.
>>>> Just reboot? Perhaps not, cause all OSD's will begin to remap/backfill, they are instructed to do so. Well, declare them as "osd lost"?
>>>> Dangerous. Is there another way I miss in doing node-maintenance? Will we have to wait for bobtail for far less hassle with all remapping and resources?
>>> 
>>> By default the monitors won't mark an OSD out in the time it takes to
>>> reboot, but if maintenance takes longer, you can drain data from the
>>> node.
>>> 
>>> A simple way to rate limit it yourself is by slowly lowering the
>>> weights of the OSDs on the host you want to update, e.g. by 0.1 at a
>>> time and waiting for recovery to complete before lowering again. Once
>>> they're at 0 and the cluster is healthy, they're not responsible for
>>> any data anymore, and the node can be rebooted.
>>> 
>> 
>> true. Should have mentioned knowing smooth way. But for a planned reboot this take way too much time ;)
>> But if it's recommended, it's recommended ;)
>> 
> 
> 
> I did rolling reboots of our whole cluster a few days ago (3.4.20). When the
> system reboots and no fsck is done, ceph won't start to backfill in my setup.
> 
> I had some nodes do fsck after upgrade so ceph marked the osd as down and
> started to backfill, but once the missing osd was back up running again, the
> backfill stopped and ceph did just a little bit of peering and was healthy in a
> few minutes again (2-5 minutes)…
> 

if you encounter all BIOS-, POST-, RAID-controller-checks, linux-boot, openvswitch-STP setup and so on, one can imagine, that a reboot takes a "couple-of-minutes", normally with our setup after 30 seconds the cluster shall detect some outage and start to do it's work.
Everytings fine, but perhaps we could avoid big load in the cluster to remap and re-remap ( "Theme: slow requests") I have to ask in means of QoS for a "better way" ;)
All that stuff had a big customer impact in the past… Time to ask.

Kind reg's

Oliver.

 

> 
> 
> 
> -- 
> 
> Mit freundlichen Grüßen,
> 
> Florian Wiessner
> 
> Smart Weblications GmbH
> Martinsberger Str. 1
> D-95119 Naila
> 
> fon.: +49 9282 9638 200
> fax.: +49 9282 9638 205
> 24/7: +49 900 144 000 00 - 0,99 EUR/Min*
> http://www.smart-weblications.de
> 
> --
> Sitz der Gesellschaft: Naila
> Geschäftsführer: Florian Wiessner
> HRB-Nr.: HRB 3840 Amtsgericht Hof
> *aus dem dt. Festnetz, ggf. abweichende Preise aus dem Mobilfunknetz
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Best practice with 0.48.2 to take a node into maintenance

[Gregory Farnum]

On Mon, Dec 3, 2012 at 12:13 PM, Oliver Francke <Oliver.Francke@filoo.de> wrote:
> if you encounter all BIOS-, POST-, RAID-controller-checks, linux-boot, openvswitch-STP setup and so on, one can imagine, that a reboot takes a "couple-of-minutes", normally with our setup after 30 seconds the cluster shall detect some outage and start to do it's work.
> Everytings fine, but perhaps we could avoid big load in the cluster to remap and re-remap ( "Theme: slow requests") I have to ask in means of QoS for a "better way" ;)
> All that stuff had a big customer impact in the past… Time to ask.

If you know you're going to be doing maintenance that might take a
while, and are going to be closely monitoring your cluster for issues,
it might be appropriate to do:
ceph osd set noout

Which will prevent any OSDs from being marked "out", and thus prevent
any migrations or backfills. You can turn it off again with
ceph osd unset noout

Of course, since this means Ceph won't do any re-replication, so
you'll need to step up your manual monitoring to compensate!
-Greg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Best practice with 0.48.2 to take a node into maintenance

[David Clarke]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/12/12 08:49, Christopher Kunz wrote:
> Am 03.12.12 20:14, schrieb Josh Durgin:
>> On 12/03/2012 11:05 AM, Oliver Francke wrote:
>>> Hi *,
>>> 
>>> well, even if 0.48.2 is really stable and reliable, it is not everytime the case with linux
>>> kernel. We have a couple of nodes, where an update would make life better. So, as our
>>> OSD-nodes have to care for VM's too, it's not the problem to let them drain so migrate all
>>> of them to other nodes. Just reboot? Perhaps not, cause all OSD's will begin to 
>>> remap/backfill, they are instructed to do so. Well, declare them as "osd lost"? Dangerous.
>>> Is there another way I miss in doing node-maintenance? Will we have to wait for bobtail for
>>> far less hassle with all remapping and resources?
>> 
>> By default the monitors won't mark an OSD out in the time it takes to reboot, but if
>> maintenance takes longer, you can drain data from the node.
> Hi,
> 
> what time is that (in seconds) and how can we reliably test this?

I believe that the time out you're referring to is:

'mon osd down out interval', which defaults to 300 seconds.

http://ceph.com/docs/master/rados/configuration/mon-config-ref/

Also, if you're concerned about the time it takes to reboot a machine (sans fsck) then you may
want to consider using something like kexec (kexec-tools package in Debian/Ubuntu).

http://en.wikipedia.org/wiki/Kexec


- -- 
David Clarke
Systems Architect
Catalyst IT
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlC9DhYACgkQRgFDJLQLJc3X2gCcDfk0rXLUXL90R4rYGNyFFLXE
hoAAnRkAMoSNc/27o6R4IGcLDX6u7Mpe
=/2TX
-----END PGP SIGNATURE-----

Re: Best practice with 0.48.2 to take a node into maintenance

[Smart Weblications GmbH - Florian Wiessner]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Am 03.12.2012 21:39, schrieb David Clarke:
> On 04/12/12 08:49, Christopher Kunz wrote:
>> Am 03.12.12 20:14, schrieb Josh Durgin:
>>> On 12/03/2012 11:05 AM, Oliver Francke wrote:
> Also, if you're concerned about the time it takes to reboot a machine (sans
> fsck) then you may want to consider using something like kexec (kexec-tools
> package in Debian/Ubuntu).
> 
> http://en.wikipedia.org/wiki/Kexec
> 

Nice idea, but the website states "Kexec is currently available on the x86
32-bit platform only."


- -- 

Mit freundlichen Grüßen,


Smart Weblications GmbH
Martinsberger Str. 1
D-95119 Naila

fon.: +49 9282 9638 200
fax.: +49 9282 9638 205
24/7: +49 900 144 000 00 - 0,99 EUR/Min*
http://www.smart-weblications.de

- --
Sitz der Gesellschaft: Naila
Geschäftsführer: Florian Wiessner
HRB-Nr.: HRB 3840 Amtsgericht Hof
*aus dem dt. Festnetz, ggf. abweichende Preise aus dem Mobilfunknetz

- -- 

Mit freundlichen Grüßen,

Florian Wiessner

Smart Weblications GmbH
Martinsberger Str. 1
D-95119 Naila

fon.: +49 9282 9638 200
fax.: +49 9282 9638 205
24/7: +49 900 144 000 00 - 0,99 EUR/Min*
http://www.smart-weblications.de

- --
Sitz der Gesellschaft: Naila
Geschäftsführer: Florian Wiessner
HRB-Nr.: HRB 3840 Amtsgericht Hof
*aus dem dt. Festnetz, ggf. abweichende Preise aus dem Mobilfunknetz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlC9EKEACgkQIDU7oJmNmZoUfACgtLyYHChlEy4hMZbyiGAFmBFb
59YAn2usmqe+fxSwXrkd5fpEE7mO5jCz
=RDiH
-----END PGP SIGNATURE-----
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Best practice with 0.48.2 to take a node into maintenance

[David Clarke]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/12/12 09:50, Smart Weblications GmbH - Florian Wiessner wrote:
> Am 03.12.2012 21:39, schrieb David Clarke:
>> On 04/12/12 08:49, Christopher Kunz wrote:
>>> Am 03.12.12 20:14, schrieb Josh Durgin:
>>>> On 12/03/2012 11:05 AM, Oliver Francke wrote:
>> Also, if you're concerned about the time it takes to reboot a machine (sans fsck) then you
>> may want to consider using something like kexec (kexec-tools package in Debian/Ubuntu).
> 
>> http://en.wikipedia.org/wiki/Kexec
> 
> 
> Nice idea, but the website states "Kexec is currently available on the x86 32-bit platform
> only."

That has to be old information - we're using it in anger on 64 bit x86 machines, from both Intel
and AMD.  I have no idea about ARM etc, however.


- -- 
David Clarke
Systems Architect
Catalyst IT
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlC9EsoACgkQRgFDJLQLJc2RUgCeNEV8vLYRq32Kvd8nzQw+sAKk
sbgAoIdfyuW53SEer1AVE/97+otdzn5b
=xHkx
-----END PGP SIGNATURE-----

Re: Best practice with 0.48.2 to take a node into maintenance

[Christopher Kunz]

Hey,
> That has to be old information - we're using it in anger on 64 bit x86
> machines, from both Intel
> and AMD.  I have no idea about ARM etc, however.
just to make absolutely sure: Are you using kexec "in anger" to switch
Ceph nodes to new kernels? I'm very doubtful that the approach will work
seamlessly with Ceph, as it is rather picky regarding kernel state...

Regards,

--ck

Re: Best practice with 0.48.2 to take a node into maintenance

[David Clarke]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/12/12 23:35, Christopher Kunz wrote:
> Hey,
>> That has to be old information - we're using it in anger on 64 bit x86 machines, from both
>> Intel and AMD.  I have no idea about ARM etc, however.
> just to make absolutely sure: Are you using kexec "in anger" to switch Ceph nodes to new
> kernels? I'm very doubtful that the approach will work seamlessly with Ceph, as it is rather
> picky regarding kernel state...

The normal shutdown cycle is activated, and Ceph on that node is stopped.  However, at the end of
the rc.6 scripts is one to, instead of rebooting, exec a new kernel and then progress as normal
through the boot process, and Ceph started again.

Haven't encountered any major issues thus far.


- -- 
David Clarke
Systems Architect
Catalyst IT
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlC+a5cACgkQRgFDJLQLJc2SOwCeKCc9aV1z09acvA3vOIMjHpJ1
F5wAn32idJCPe4WNcu0smSNzUSZv634F
=JBvD
-----END PGP SIGNATURE-----