handling fs errors

handling fs errors

[Sage Weil]

We observed an interesting situation over the weekend.  The XFS volume 
ceph-osd locked up (hung in xfs_ilock) for somewhere between 2 and 4 
minutes.  After 3 minutes (180s), ceph-osd gave up waiting and committed 
suicide.  XFS seemed to unwedge itself a bit after that, as the daemon was 
able to restart and continue.

The problem is that during that 180s the OSD was claiming to be alive but 
not able to do any IO.  That heartbeat check is meant as a sanity check 
against a wedged kernel, but waiting so long meant that the ceph-osd 
wasn't failed by the cluster quickly enough and client IO stalled.

We could simply change that timeout to something close to the heartbeat 
interval (currently default is 20s).  That will make ceph-osd much more 
sensitive to fs stalls that may be transient (high load, whatever).

Another option would be to make the osd heartbeat replies conditional on 
whether the internal heartbeat is healthy.  Then the heartbeat warnings 
could start at 10-20s, ping replies would pause, but the suicide could 
still be 180s out.  If the stall is short-lived, pings will continue, the 
osd will mark itself back up (if it was marked down) and continue.

Having written that out, the last option sounds like the obvious choice.  
Any other thoughts?

sage

Re: handling fs errors

[Yehuda Sadeh]

On Mon, Jan 21, 2013 at 10:05 PM, Sage Weil <sage@inktank.com> wrote:
> We observed an interesting situation over the weekend.  The XFS volume
> ceph-osd locked up (hung in xfs_ilock) for somewhere between 2 and 4
> minutes.  After 3 minutes (180s), ceph-osd gave up waiting and committed
> suicide.  XFS seemed to unwedge itself a bit after that, as the daemon was
> able to restart and continue.
>
> The problem is that during that 180s the OSD was claiming to be alive but
> not able to do any IO.  That heartbeat check is meant as a sanity check
> against a wedged kernel, but waiting so long meant that the ceph-osd
> wasn't failed by the cluster quickly enough and client IO stalled.
>
> We could simply change that timeout to something close to the heartbeat
> interval (currently default is 20s).  That will make ceph-osd much more
> sensitive to fs stalls that may be transient (high load, whatever).
>
> Another option would be to make the osd heartbeat replies conditional on
> whether the internal heartbeat is healthy.  Then the heartbeat warnings
> could start at 10-20s, ping replies would pause, but the suicide could
> still be 180s out.  If the stall is short-lived, pings will continue, the
> osd will mark itself back up (if it was marked down) and continue.
>
> Having written that out, the last option sounds like the obvious choice.
> Any other thoughts?
>

Another option would be to have the osd reply to the ping with some
health description.

Yehuda

Re: handling fs errors

[Andrey Korolyov]

On Tue, Jan 22, 2013 at 10:05 AM, Sage Weil <sage@inktank.com> wrote:
> We observed an interesting situation over the weekend.  The XFS volume
> ceph-osd locked up (hung in xfs_ilock) for somewhere between 2 and 4
> minutes.  After 3 minutes (180s), ceph-osd gave up waiting and committed
> suicide.  XFS seemed to unwedge itself a bit after that, as the daemon was
> able to restart and continue.
>
> The problem is that during that 180s the OSD was claiming to be alive but
> not able to do any IO.  That heartbeat check is meant as a sanity check
> against a wedged kernel, but waiting so long meant that the ceph-osd
> wasn't failed by the cluster quickly enough and client IO stalled.
>
> We could simply change that timeout to something close to the heartbeat
> interval (currently default is 20s).  That will make ceph-osd much more
> sensitive to fs stalls that may be transient (high load, whatever).
>
> Another option would be to make the osd heartbeat replies conditional on
> whether the internal heartbeat is healthy.  Then the heartbeat warnings
> could start at 10-20s, ping replies would pause, but the suicide could
> still be 180s out.  If the stall is short-lived, pings will continue, the
> osd will mark itself back up (if it was marked down) and continue.
>
> Having written that out, the last option sounds like the obvious choice.
> Any other thoughts?
>
> sage

Seems to be possible to run in domino-style failing marks there if
lock is triggered frequently enough and depends only on pure amount of
workload. By the way, was that fs aged or you`re able to catch the
lock on fresh one? And which kernel you have run there?

Thanks!

> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: handling fs errors

[Chen, Xiaoxi]

Is there any known connection with the previous discussion " Hit suicide timeout after adding new osd" or "Ceph unstable on XFS" ?

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
Sent: 2013年1月22日 14:06
To: ceph-devel@vger.kernel.org
Subject: handling fs errors

We observed an interesting situation over the weekend.  The XFS volume ceph-osd locked up (hung in xfs_ilock) for somewhere between 2 and 4 minutes.  After 3 minutes (180s), ceph-osd gave up waiting and committed suicide.  XFS seemed to unwedge itself a bit after that, as the daemon was able to restart and continue.

The problem is that during that 180s the OSD was claiming to be alive but not able to do any IO.  That heartbeat check is meant as a sanity check against a wedged kernel, but waiting so long meant that the ceph-osd wasn't failed by the cluster quickly enough and client IO stalled.

We could simply change that timeout to something close to the heartbeat interval (currently default is 20s).  That will make ceph-osd much more sensitive to fs stalls that may be transient (high load, whatever).

Another option would be to make the osd heartbeat replies conditional on whether the internal heartbeat is healthy.  Then the heartbeat warnings could start at 10-20s, ping replies would pause, but the suicide could still be 180s out.  If the stall is short-lived, pings will continue, the osd will mark itself back up (if it was marked down) and continue.

Having written that out, the last option sounds like the obvious choice.  
Any other thoughts?

sage
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: handling fs errors

[Wido den Hollander]

On 01/22/2013 07:12 AM, Yehuda Sadeh wrote:
> On Mon, Jan 21, 2013 at 10:05 PM, Sage Weil <sage@inktank.com> wrote:
>> We observed an interesting situation over the weekend.  The XFS volume
>> ceph-osd locked up (hung in xfs_ilock) for somewhere between 2 and 4
>> minutes.  After 3 minutes (180s), ceph-osd gave up waiting and committed
>> suicide.  XFS seemed to unwedge itself a bit after that, as the daemon was
>> able to restart and continue.
>>
>> The problem is that during that 180s the OSD was claiming to be alive but
>> not able to do any IO.  That heartbeat check is meant as a sanity check
>> against a wedged kernel, but waiting so long meant that the ceph-osd
>> wasn't failed by the cluster quickly enough and client IO stalled.
>>
>> We could simply change that timeout to something close to the heartbeat
>> interval (currently default is 20s).  That will make ceph-osd much more
>> sensitive to fs stalls that may be transient (high load, whatever).
>>
>> Another option would be to make the osd heartbeat replies conditional on
>> whether the internal heartbeat is healthy.  Then the heartbeat warnings
>> could start at 10-20s, ping replies would pause, but the suicide could
>> still be 180s out.  If the stall is short-lived, pings will continue, the
>> osd will mark itself back up (if it was marked down) and continue.
>>
>> Having written that out, the last option sounds like the obvious choice.
>> Any other thoughts?
>>
>
> Another option would be to have the osd reply to the ping with some
> health description.
>

Looking to the future with more monitoring that might be a good idea.

If an OSD simply stops sending heartbeats if the internal conditions 
aren't met you don't know what's going on.

If the heartbeat would have metadata which tells: "I'm here, but not in 
such a good shape" that could be reported back to the monitors.

Monitoring tools could read this out and could sent out 
notifications/alerts to where they want.

Now we assume I/O completely stalls, but the metadata could also contain 
high latency? If the latency goes over threshold X you can still mark 
the OSD out temporarily since it will impact clients, but some 
information towards the monitor might be useful.

Wido

> Yehuda
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: handling fs errors

[Gregory Farnum]

On Tuesday, January 22, 2013 at 5:12 AM, Wido den Hollander wrote:
> On 01/22/2013 07:12 AM, Yehuda Sadeh wrote:
> > On Mon, Jan 21, 2013 at 10:05 PM, Sage Weil <sage@inktank.com (mailto:sage@inktank.com)> wrote:
> > > We observed an interesting situation over the weekend. The XFS volume
> > > ceph-osd locked up (hung in xfs_ilock) for somewhere between 2 and 4
> > > minutes. After 3 minutes (180s), ceph-osd gave up waiting and committed
> > > suicide. XFS seemed to unwedge itself a bit after that, as the daemon was
> > > able to restart and continue.
> > > 
> > > The problem is that during that 180s the OSD was claiming to be alive but
> > > not able to do any IO. That heartbeat check is meant as a sanity check
> > > against a wedged kernel, but waiting so long meant that the ceph-osd
> > > wasn't failed by the cluster quickly enough and client IO stalled.
> > > 
> > > We could simply change that timeout to something close to the heartbeat
> > > interval (currently default is 20s). That will make ceph-osd much more
> > > sensitive to fs stalls that may be transient (high load, whatever).
> > > 
> > > Another option would be to make the osd heartbeat replies conditional on
> > > whether the internal heartbeat is healthy. Then the heartbeat warnings
> > > could start at 10-20s, ping replies would pause, but the suicide could
> > > still be 180s out. If the stall is short-lived, pings will continue, the
> > > osd will mark itself back up (if it was marked down) and continue.
> > > 
> > > Having written that out, the last option sounds like the obvious choice.
> > > Any other thoughts?
> > 
> > 
> > 
> > Another option would be to have the osd reply to the ping with some
> > health description.
> 
> 
> 
> Looking to the future with more monitoring that might be a good idea.
> 
> If an OSD simply stops sending heartbeats if the internal conditions 
> aren't met you don't know what's going on.
> 
> If the heartbeat would have metadata which tells: "I'm here, but not in 
> such a good shape" that could be reported back to the monitors.


I think we want to move towards more comprehensive pinging like this, but it's not something to do in haste. Pausing pings when the internal threads are disappearing sounds like a good simple step to make the reporting better match reality.
-Greg 

Re: handling fs errors

[Dimitri Maziuk]

[-- Attachment #1: Type: text/plain, Size: 649 bytes --]

On 01/22/2013 12:05 AM, Sage Weil wrote:
> We observed an interesting situation over the weekend.  The XFS volume 
> ceph-osd locked up (hung in xfs_ilock) for somewhere between 2 and 4 
> minutes.
...

FWIW I see this often enough on cheap sata drives: they've a failure
mode that makes sata driver timeout, reset the link, resend the command,
rinse, lather, repeat. (You usually get "slow to respond, please be
patient" and/or "resetting link" in syslog & console.) It's at the low
enough level to freeze the whole system for minutes.

-- 
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 254 bytes --]

Re: handling fs errors

[Andrey Korolyov]

On Tue, Jan 22, 2013 at 10:05 AM, Sage Weil <sage@inktank.com> wrote:
> We observed an interesting situation over the weekend.  The XFS volume
> ceph-osd locked up (hung in xfs_ilock) for somewhere between 2 and 4
> minutes.  After 3 minutes (180s), ceph-osd gave up waiting and committed
> suicide.  XFS seemed to unwedge itself a bit after that, as the daemon was
> able to restart and continue.
>
> The problem is that during that 180s the OSD was claiming to be alive but
> not able to do any IO.  That heartbeat check is meant as a sanity check
> against a wedged kernel, but waiting so long meant that the ceph-osd
> wasn't failed by the cluster quickly enough and client IO stalled.
>
> We could simply change that timeout to something close to the heartbeat
> interval (currently default is 20s).  That will make ceph-osd much more
> sensitive to fs stalls that may be transient (high load, whatever).
>
> Another option would be to make the osd heartbeat replies conditional on
> whether the internal heartbeat is healthy.  Then the heartbeat warnings
> could start at 10-20s, ping replies would pause, but the suicide could
> still be 180s out.  If the stall is short-lived, pings will continue, the
> osd will mark itself back up (if it was marked down) and continue.
>
> Having written that out, the last option sounds like the obvious choice.
> Any other thoughts?
>
> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

By the way, is there any worth of preventing situation below - one
host buried down in the softlocks marking almost all neighbors down
for a short time? That`s harmless, because all osds will rejoin
cluster in couple of next heartbeats, but all I/O will stuck at this
time, rather than only operations with pgs on failing osd, so may be
it`ll be useful to introduce kind of down-mark-quorum for such cases.

2013-01-22 14:40:31.481174 mon.0 [INF] osd.0 10.5.0.10:6800/6578
failed (3 reports from 1 peers after 2013-01-22 14:40:54.481085 >=
grace 20
.000000)
2013-01-22 14:40:31.481293 mon.0 [INF] osd.1 10.5.0.11:6800/6488
failed (3 reports from 1 peers after 2013-01-22 14:40:54.481228 >=
grace 20
.000000)
2013-01-22 14:40:31.481410 mon.0 [INF] osd.2 10.5.0.12:6803/7561
failed (3 reports from 1 peers after 2013-01-22 14:40:54.481355 >=
grace 20
.000000)
2013-01-22 14:40:31.481522 mon.0 [INF] osd.4 10.5.0.14:6803/5697
failed (3 reports from 1 peers after 2013-01-22 14:40:54.481467 >=
grace 20
.000000)
2013-01-22 14:40:31.481641 mon.0 [INF] osd.6 10.5.0.16:6803/5679
failed (3 reports from 1 peers after 2013-01-22 14:40:54.481586 >=
grace 20.000000)
2013-01-22 14:40:31.481746 mon.0 [INF] osd.8 10.5.0.10:6803/6638
failed (3 reports from 1 peers after 2013-01-22 14:40:54.481700 >=
grace 20.000000)
2013-01-22 14:40:31.481863 mon.0 [INF] osd.9 10.5.0.11:6803/6547
failed (3 reports from 1 peers after 2013-01-22 14:40:54.481811 >=
grace 20.000000)
2013-01-22 14:40:31.481976 mon.0 [INF] osd.10 10.5.0.12:6800/7019
failed (3 reports from 1 peers after 2013-01-22 14:40:54.481916 >=
grace 20.000000)
2013-01-22 14:40:31.482077 mon.0 [INF] osd.12 10.5.0.14:6800/5637
failed (3 reports from 1 peers after 2013-01-22 14:40:54.482022 >=
grace 20.000000)
2013-01-22 14:40:31.482184 mon.0 [INF] osd.14 10.5.0.16:6800/5620
failed (3 reports from 1 peers after 2013-01-22 14:40:54.482130 >=
grace 20.000000)
2013-01-22 14:40:31.482334 mon.0 [INF] osd.17 10.5.0.31:6800/5854
failed (3 reports from 1 peers after 2013-01-22 14:40:54.482275 >=
grace 20.000000)
2013-01-22 14:40:31.482436 mon.0 [INF] osd.18 10.5.0.32:6800/5981
failed (3 reports from 1 peers after 2013-01-22 14:40:54.482389 >=
grace 20.000000)
2013-01-22 14:40:31.482539 mon.0 [INF] osd.19 10.5.0.33:6800/5570
failed (3 reports from 1 peers after 2013-01-22 14:40:54.482489 >=
grace 20.000000)
2013-01-22 14:40:31.482667 mon.0 [INF] osd.20 10.5.0.34:6800/5643
failed (3 reports from 1 peers after 2013-01-22 14:40:54.482620 >=
grace 20.000000)

Re: handling fs errors

[Sage Weil]

On Wed, 23 Jan 2013, Andrey Korolyov wrote:
> On Tue, Jan 22, 2013 at 10:05 AM, Sage Weil <sage@inktank.com> wrote:
> > We observed an interesting situation over the weekend.  The XFS volume
> > ceph-osd locked up (hung in xfs_ilock) for somewhere between 2 and 4
> > minutes.  After 3 minutes (180s), ceph-osd gave up waiting and committed
> > suicide.  XFS seemed to unwedge itself a bit after that, as the daemon was
> > able to restart and continue.
> >
> > The problem is that during that 180s the OSD was claiming to be alive but
> > not able to do any IO.  That heartbeat check is meant as a sanity check
> > against a wedged kernel, but waiting so long meant that the ceph-osd
> > wasn't failed by the cluster quickly enough and client IO stalled.
> >
> > We could simply change that timeout to something close to the heartbeat
> > interval (currently default is 20s).  That will make ceph-osd much more
> > sensitive to fs stalls that may be transient (high load, whatever).
> >
> > Another option would be to make the osd heartbeat replies conditional on
> > whether the internal heartbeat is healthy.  Then the heartbeat warnings
> > could start at 10-20s, ping replies would pause, but the suicide could
> > still be 180s out.  If the stall is short-lived, pings will continue, the
> > osd will mark itself back up (if it was marked down) and continue.
> >
> > Having written that out, the last option sounds like the obvious choice.
> > Any other thoughts?
> >
> > sage
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> By the way, is there any worth of preventing situation below - one
> host buried down in the softlocks marking almost all neighbors down
> for a short time? That`s harmless, because all osds will rejoin
> cluster in couple of next heartbeats, but all I/O will stuck at this
> time, rather than only operations with pgs on failing osd, so may be
> it`ll be useful to introduce kind of down-mark-quorum for such cases.
> 
> 2013-01-22 14:40:31.481174 mon.0 [INF] osd.0 10.5.0.10:6800/6578
> failed (3 reports from 1 peers after 2013-01-22 14:40:54.481085 >=
> grace 20
> .000000)

I believe there is already a tuanble to adjust this... 'min reporters' or 
something, check 'ceph --show-config | grep ^mon'.

sage


> 2013-01-22 14:40:31.481293 mon.0 [INF] osd.1 10.5.0.11:6800/6488
> failed (3 reports from 1 peers after 2013-01-22 14:40:54.481228 >=
> grace 20
> .000000)
> 2013-01-22 14:40:31.481410 mon.0 [INF] osd.2 10.5.0.12:6803/7561
> failed (3 reports from 1 peers after 2013-01-22 14:40:54.481355 >=
> grace 20
> .000000)
> 2013-01-22 14:40:31.481522 mon.0 [INF] osd.4 10.5.0.14:6803/5697
> failed (3 reports from 1 peers after 2013-01-22 14:40:54.481467 >=
> grace 20
> .000000)
> 2013-01-22 14:40:31.481641 mon.0 [INF] osd.6 10.5.0.16:6803/5679
> failed (3 reports from 1 peers after 2013-01-22 14:40:54.481586 >=
> grace 20.000000)
> 2013-01-22 14:40:31.481746 mon.0 [INF] osd.8 10.5.0.10:6803/6638
> failed (3 reports from 1 peers after 2013-01-22 14:40:54.481700 >=
> grace 20.000000)
> 2013-01-22 14:40:31.481863 mon.0 [INF] osd.9 10.5.0.11:6803/6547
> failed (3 reports from 1 peers after 2013-01-22 14:40:54.481811 >=
> grace 20.000000)
> 2013-01-22 14:40:31.481976 mon.0 [INF] osd.10 10.5.0.12:6800/7019
> failed (3 reports from 1 peers after 2013-01-22 14:40:54.481916 >=
> grace 20.000000)
> 2013-01-22 14:40:31.482077 mon.0 [INF] osd.12 10.5.0.14:6800/5637
> failed (3 reports from 1 peers after 2013-01-22 14:40:54.482022 >=
> grace 20.000000)
> 2013-01-22 14:40:31.482184 mon.0 [INF] osd.14 10.5.0.16:6800/5620
> failed (3 reports from 1 peers after 2013-01-22 14:40:54.482130 >=
> grace 20.000000)
> 2013-01-22 14:40:31.482334 mon.0 [INF] osd.17 10.5.0.31:6800/5854
> failed (3 reports from 1 peers after 2013-01-22 14:40:54.482275 >=
> grace 20.000000)
> 2013-01-22 14:40:31.482436 mon.0 [INF] osd.18 10.5.0.32:6800/5981
> failed (3 reports from 1 peers after 2013-01-22 14:40:54.482389 >=
> grace 20.000000)
> 2013-01-22 14:40:31.482539 mon.0 [INF] osd.19 10.5.0.33:6800/5570
> failed (3 reports from 1 peers after 2013-01-22 14:40:54.482489 >=
> grace 20.000000)
> 2013-01-22 14:40:31.482667 mon.0 [INF] osd.20 10.5.0.34:6800/5643
> failed (3 reports from 1 peers after 2013-01-22 14:40:54.482620 >=
> grace 20.000000)
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: handling fs errors

[Sage Weil]

On Tue, 22 Jan 2013, Dimitri Maziuk wrote:
> On 01/22/2013 12:05 AM, Sage Weil wrote:
> > We observed an interesting situation over the weekend.  The XFS volume 
> > ceph-osd locked up (hung in xfs_ilock) for somewhere between 2 and 4 
> > minutes.
> ...
> 
> FWIW I see this often enough on cheap sata drives: they've a failure
> mode that makes sata driver timeout, reset the link, resend the command,
> rinse, lather, repeat. (You usually get "slow to respond, please be
> patient" and/or "resetting link" in syslog & console.) It's at the low
> enough level to freeze the whole system for minutes.

Excellent point, I'd forgotten about that.

This task is queued up for this sprint, by the way, and marked for 
backport to bobtail.

	http://tracker.newdream.net/issues/3888

Thanks-
sage