* automatic repair of inconsistent pg?
@ 2012-12-25 2:01 Stefan Priebe
2012-12-25 6:54 ` Sage Weil
0 siblings, 1 reply; 7+ messages in thread
From: Stefan Priebe @ 2012-12-25 2:01 UTC (permalink / raw)
To: ceph-devel@vger.kernel.org
Hello list,
today i got the following ceph status output:
2012-12-25 02:57:00.632945 mon.0 [INF] pgmap v1394388: 7632 pgs: 7631
active+clean, 1 active+clean+inconsistent; 151 GB data, 307 GB used,
5028 GB / 5336 GB avail
i then grepped the inconsistent pg by:
# ceph pg dump - | grep inconsistent
3.ccf 10 0 0 0 41037824 155930 155930
active+clean+inconsistent 2012-12-25 01:51:35.318459 6243'2107
6190'9847 [14,42] [14,42] 6243'2107 2012-12-25
01:51:35.318436 6007'2074 2012-12-23 01:51:24.386366
and initiated a repair:
# ceph pg repair 3.ccf
instructing pg 3.ccf on osd.14 to repair
The log output then was:
2012-12-25 02:56:59.056382 osd.14 [ERR] 3.ccf osd.42 missing
1c602ccf/rbd_data.4904d6b8b4567.0000000000000b84/head//3
2012-12-25 02:56:59.056385 osd.14 [ERR] 3.ccf osd.42 missing
ceb55ccf/rbd_data.48cc66b8b4567.0000000000001538/head//3
2012-12-25 02:56:59.097989 osd.14 [ERR] 3.ccf osd.42 missing
dba6bccf/rbd_data.4797d6b8b4567.00000000000015ad/head//3
2012-12-25 02:56:59.097991 osd.14 [ERR] 3.ccf osd.42 missing
a4deccf/rbd_data.45f956b8b4567.00000000000003d5/head//3
2012-12-25 02:56:59.098022 osd.14 [ERR] 3.ccf repair 4 missing, 0
inconsistent objects
2012-12-25 02:56:59.098046 osd.14 [ERR] 3.ccf repair 4 errors, 4 fixed
Why doesn't ceph repair this automatically? Ho could this happen at all?
Stefan
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: automatic repair of inconsistent pg?
2012-12-25 2:01 automatic repair of inconsistent pg? Stefan Priebe
@ 2012-12-25 6:54 ` Sage Weil
2012-12-30 18:22 ` Samuel Just
[not found] ` <CA+4uBUbDVC0pKEfGfHmEuaVvZsoHWyZrxoE+vrFSddgWLGeELQ@mail.gmail.com>
0 siblings, 2 replies; 7+ messages in thread
From: Sage Weil @ 2012-12-25 6:54 UTC (permalink / raw)
To: Stefan Priebe; +Cc: ceph-devel@vger.kernel.org
On Tue, 25 Dec 2012, Stefan Priebe wrote:
> Hello list,
>
> today i got the following ceph status output:
> 2012-12-25 02:57:00.632945 mon.0 [INF] pgmap v1394388: 7632 pgs: 7631
> active+clean, 1 active+clean+inconsistent; 151 GB data, 307 GB used, 5028 GB /
> 5336 GB avail
>
>
> i then grepped the inconsistent pg by:
> # ceph pg dump - | grep inconsistent
> 3.ccf 10 0 0 0 41037824 155930 155930
> active+clean+inconsistent 2012-12-25 01:51:35.318459 6243'2107
> 6190'9847 [14,42] [14,42] 6243'2107 2012-12-25 01:51:35.318436
> 6007'2074 2012-12-23 01:51:24.386366
>
> and initiated a repair:
> # ceph pg repair 3.ccf
> instructing pg 3.ccf on osd.14 to repair
>
> The log output then was:
> 2012-12-25 02:56:59.056382 osd.14 [ERR] 3.ccf osd.42 missing
> 1c602ccf/rbd_data.4904d6b8b4567.0000000000000b84/head//3
> 2012-12-25 02:56:59.056385 osd.14 [ERR] 3.ccf osd.42 missing
> ceb55ccf/rbd_data.48cc66b8b4567.0000000000001538/head//3
> 2012-12-25 02:56:59.097989 osd.14 [ERR] 3.ccf osd.42 missing
> dba6bccf/rbd_data.4797d6b8b4567.00000000000015ad/head//3
> 2012-12-25 02:56:59.097991 osd.14 [ERR] 3.ccf osd.42 missing
> a4deccf/rbd_data.45f956b8b4567.00000000000003d5/head//3
> 2012-12-25 02:56:59.098022 osd.14 [ERR] 3.ccf repair 4 missing, 0 inconsistent
> objects
> 2012-12-25 02:56:59.098046 osd.14 [ERR] 3.ccf repair 4 errors, 4 fixed
>
> Why doesn't ceph repair this automatically? Ho could this happen at all?
We just made some fixes to repair in next (it was broken sometime between
~0.53 and 0.55). The latest next should repair it. In general we don't
repair automatically lest we inadvertantly propagate bad data or paper
over a bug.
As for the original source of the missing objects... I'm not sure. There
were some fixed races related to backfill that could lead to an object
being missed, but Sam would know more about how likely that actually is.
sage
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: automatic repair of inconsistent pg?
2012-12-25 6:54 ` Sage Weil
@ 2012-12-30 18:22 ` Samuel Just
[not found] ` <CA+4uBUbDVC0pKEfGfHmEuaVvZsoHWyZrxoE+vrFSddgWLGeELQ@mail.gmail.com>
1 sibling, 0 replies; 7+ messages in thread
From: Samuel Just @ 2012-12-30 18:22 UTC (permalink / raw)
To: Sage Weil; +Cc: Stefan Priebe, ceph-devel@vger.kernel.org
This is somewhat more likely to have been a bug in the replication
logic (there were a few fixed between 0.53 and 0.55). Had there been
any recent osd failures?
-Sam
On Mon, Dec 24, 2012 at 10:54 PM, Sage Weil <sage@inktank.com> wrote:
> On Tue, 25 Dec 2012, Stefan Priebe wrote:
>> Hello list,
>>
>> today i got the following ceph status output:
>> 2012-12-25 02:57:00.632945 mon.0 [INF] pgmap v1394388: 7632 pgs: 7631
>> active+clean, 1 active+clean+inconsistent; 151 GB data, 307 GB used, 5028 GB /
>> 5336 GB avail
>>
>>
>> i then grepped the inconsistent pg by:
>> # ceph pg dump - | grep inconsistent
>> 3.ccf 10 0 0 0 41037824 155930 155930
>> active+clean+inconsistent 2012-12-25 01:51:35.318459 6243'2107
>> 6190'9847 [14,42] [14,42] 6243'2107 2012-12-25 01:51:35.318436
>> 6007'2074 2012-12-23 01:51:24.386366
>>
>> and initiated a repair:
>> # ceph pg repair 3.ccf
>> instructing pg 3.ccf on osd.14 to repair
>>
>> The log output then was:
>> 2012-12-25 02:56:59.056382 osd.14 [ERR] 3.ccf osd.42 missing
>> 1c602ccf/rbd_data.4904d6b8b4567.0000000000000b84/head//3
>> 2012-12-25 02:56:59.056385 osd.14 [ERR] 3.ccf osd.42 missing
>> ceb55ccf/rbd_data.48cc66b8b4567.0000000000001538/head//3
>> 2012-12-25 02:56:59.097989 osd.14 [ERR] 3.ccf osd.42 missing
>> dba6bccf/rbd_data.4797d6b8b4567.00000000000015ad/head//3
>> 2012-12-25 02:56:59.097991 osd.14 [ERR] 3.ccf osd.42 missing
>> a4deccf/rbd_data.45f956b8b4567.00000000000003d5/head//3
>> 2012-12-25 02:56:59.098022 osd.14 [ERR] 3.ccf repair 4 missing, 0 inconsistent
>> objects
>> 2012-12-25 02:56:59.098046 osd.14 [ERR] 3.ccf repair 4 errors, 4 fixed
>>
>> Why doesn't ceph repair this automatically? Ho could this happen at all?
>
> We just made some fixes to repair in next (it was broken sometime between
> ~0.53 and 0.55). The latest next should repair it. In general we don't
> repair automatically lest we inadvertantly propagate bad data or paper
> over a bug.
>
> As for the original source of the missing objects... I'm not sure. There
> were some fixed races related to backfill that could lead to an object
> being missed, but Sam would know more about how likely that actually is.
>
> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: automatic repair of inconsistent pg?
[not found] ` <CA+4uBUbDVC0pKEfGfHmEuaVvZsoHWyZrxoE+vrFSddgWLGeELQ@mail.gmail.com>
@ 2012-12-30 19:28 ` Stefan Priebe
[not found] ` <CA+4uBUb81Gv-4vjKTd8UvV7V8Ep7PG3roug30+0e9hHksTc35g@mail.gmail.com>
0 siblings, 1 reply; 7+ messages in thread
From: Stefan Priebe @ 2012-12-30 19:28 UTC (permalink / raw)
To: Samuel Just; +Cc: Sage Weil, ceph-devel
Am 30.12.2012 19:17, schrieb Samuel Just:
> This is somewhat more likely to have been a bug in the replication logic
> (there were a few fixed between 0.53 and 0.55). Had there been any
> recent osd failures?
Yes i was stressing CEPH with failures (power, link, disk, ...).
Stefan
> On Dec 24, 2012 10:55 PM, "Sage Weil" <sage@inktank.com
> <mailto:sage@inktank.com>> wrote:
>
> On Tue, 25 Dec 2012, Stefan Priebe wrote:
> > Hello list,
> >
> > today i got the following ceph status output:
> > 2012-12-25 02:57:00.632945 mon.0 [INF] pgmap v1394388: 7632 pgs: 7631
> > active+clean, 1 active+clean+inconsistent; 151 GB data, 307 GB
> used, 5028 GB /
> > 5336 GB avail
> >
> >
> > i then grepped the inconsistent pg by:
> > # ceph pg dump - | grep inconsistent
> > 3.ccf 10 0 0 0 41037824 155930
> 155930
> > active+clean+inconsistent 2012-12-25 01:51:35.318459 6243'2107
> > 6190'9847 [14,42] [14,42] 6243'2107 2012-12-25
> 01:51:35.318436
> > 6007'2074 2012-12-23 01:51:24.386366
> >
> > and initiated a repair:
> > # ceph pg repair 3.ccf
> > instructing pg 3.ccf on osd.14 to repair
> >
> > The log output then was:
> > 2012-12-25 02:56:59.056382 osd.14 [ERR] 3.ccf osd.42 missing
> > 1c602ccf/rbd_data.4904d6b8b4567.0000000000000b84/head//3
> > 2012-12-25 02:56:59.056385 osd.14 [ERR] 3.ccf osd.42 missing
> > ceb55ccf/rbd_data.48cc66b8b4567.0000000000001538/head//3
> > 2012-12-25 02:56:59.097989 osd.14 [ERR] 3.ccf osd.42 missing
> > dba6bccf/rbd_data.4797d6b8b4567.00000000000015ad/head//3
> > 2012-12-25 02:56:59.097991 osd.14 [ERR] 3.ccf osd.42 missing
> > a4deccf/rbd_data.45f956b8b4567.00000000000003d5/head//3
> > 2012-12-25 02:56:59.098022 osd.14 [ERR] 3.ccf repair 4 missing, 0
> inconsistent
> > objects
> > 2012-12-25 02:56:59.098046 osd.14 [ERR] 3.ccf repair 4 errors, 4
> fixed
> >
> > Why doesn't ceph repair this automatically? Ho could this happen
> at all?
>
> We just made some fixes to repair in next (it was broken sometime
> between
> ~0.53 and 0.55). The latest next should repair it. In general we don't
> repair automatically lest we inadvertantly propagate bad data or paper
> over a bug.
>
> As for the original source of the missing objects... I'm not sure.
> There
> were some fixed races related to backfill that could lead to an object
> being missed, but Sam would know more about how likely that actually is.
>
> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> <mailto:majordomo@vger.kernel.org>
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: automatic repair of inconsistent pg?
[not found] ` <CA+4uBUb81Gv-4vjKTd8UvV7V8Ep7PG3roug30+0e9hHksTc35g@mail.gmail.com>
@ 2012-12-31 11:59 ` Stefan Priebe
2012-12-31 19:21 ` Samuel Just
0 siblings, 1 reply; 7+ messages in thread
From: Stefan Priebe @ 2012-12-31 11:59 UTC (permalink / raw)
To: Samuel Just; +Cc: ceph-devel@vger.kernel.org, Sage Weil
Am 31.12.2012 02:10, schrieb Samuel Just:
> Are you using xfs? If so, what mount options?
Yes,
noatime,nodiratime,nobarrier,logbufs=8,logbsize=256k
Stefan
>
> On Dec 30, 2012 1:28 PM, "Stefan Priebe" <s.priebe@profihost.ag
> <mailto:s.priebe@profihost.ag>> wrote:
> >
> > Am 30.12.2012 19:17, schrieb Samuel Just:
> >>
> >> This is somewhat more likely to have been a bug in the replication logic
> >> (there were a few fixed between 0.53 and 0.55). Had there been any
> >> recent osd failures?
> >
> > Yes i was stressing CEPH with failures (power, link, disk, ...).
> >
> > Stefan
> >
> >> On Dec 24, 2012 10:55 PM, "Sage Weil" <sage@inktank.com
> <mailto:sage@inktank.com>
> >> <mailto:sage@inktank.com <mailto:sage@inktank.com>>> wrote:
> >>
> >> On Tue, 25 Dec 2012, Stefan Priebe wrote:
> >> > Hello list,
> >> >
> >> > today i got the following ceph status output:
> >> > 2012-12-25 02:57:00.632945 mon.0 [INF] pgmap v1394388: 7632
> pgs: 7631
> >> > active+clean, 1 active+clean+inconsistent; 151 GB data, 307 GB
> >> used, 5028 GB /
> >> > 5336 GB avail
> >> >
> >> >
> >> > i then grepped the inconsistent pg by:
> >> > # ceph pg dump - | grep inconsistent
> >> > 3.ccf 10 0 0 0 41037824 155930
> >> 155930
> >> > active+clean+inconsistent 2012-12-25 01:51:35.318459
> 6243'2107
> >> > 6190'9847 [14,42] [14,42] 6243'2107 2012-12-25
> >> 01:51:35.318436
> >> > 6007'2074 2012-12-23 01:51:24.386366
> >> >
> >> > and initiated a repair:
> >> > # ceph pg repair 3.ccf
> >> > instructing pg 3.ccf on osd.14 to repair
> >> >
> >> > The log output then was:
> >> > 2012-12-25 02:56:59.056382 osd.14 [ERR] 3.ccf osd.42 missing
> >> > 1c602ccf/rbd_data.4904d6b8b4567.0000000000000b84/head//3
> >> > 2012-12-25 02:56:59.056385 osd.14 [ERR] 3.ccf osd.42 missing
> >> > ceb55ccf/rbd_data.48cc66b8b4567.0000000000001538/head//3
> >> > 2012-12-25 02:56:59.097989 osd.14 [ERR] 3.ccf osd.42 missing
> >> > dba6bccf/rbd_data.4797d6b8b4567.00000000000015ad/head//3
> >> > 2012-12-25 02:56:59.097991 osd.14 [ERR] 3.ccf osd.42 missing
> >> > a4deccf/rbd_data.45f956b8b4567.00000000000003d5/head//3
> >> > 2012-12-25 02:56:59.098022 osd.14 [ERR] 3.ccf repair 4 missing, 0
> >> inconsistent
> >> > objects
> >> > 2012-12-25 02:56:59.098046 osd.14 [ERR] 3.ccf repair 4 errors, 4
> >> fixed
> >> >
> >> > Why doesn't ceph repair this automatically? Ho could this happen
> >> at all?
> >>
> >> We just made some fixes to repair in next (it was broken sometime
> >> between
> >> ~0.53 and 0.55). The latest next should repair it. In general
> we don't
> >> repair automatically lest we inadvertantly propagate bad data or
> paper
> >> over a bug.
> >>
> >> As for the original source of the missing objects... I'm not sure.
> >> There
> >> were some fixed races related to backfill that could lead to an
> object
> >> being missed, but Sam would know more about how likely that
> actually is.
> >>
> >> sage
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe
> ceph-devel" in
> >> the body of a message to majordomo@vger.kernel.org
> <mailto:majordomo@vger.kernel.org>
> >> <mailto:majordomo@vger.kernel.org
> <mailto:majordomo@vger.kernel.org>>
> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: automatic repair of inconsistent pg?
2012-12-31 11:59 ` Stefan Priebe
@ 2012-12-31 19:21 ` Samuel Just
2013-01-01 20:12 ` Stefan Priebe
0 siblings, 1 reply; 7+ messages in thread
From: Samuel Just @ 2012-12-31 19:21 UTC (permalink / raw)
To: Stefan Priebe; +Cc: ceph-devel@vger.kernel.org, Sage Weil
The ceph-osd relies on fs barriers for correctness. You will want to
remove the nobarrier option to prevent future corruption.
-Sam
On Mon, Dec 31, 2012 at 3:59 AM, Stefan Priebe <s.priebe@profihost.ag> wrote:
> Am 31.12.2012 02:10, schrieb Samuel Just:
>
>> Are you using xfs? If so, what mount options?
>
>
> Yes,
> noatime,nodiratime,nobarrier,logbufs=8,logbsize=256k
>
> Stefan
>
>>
>> On Dec 30, 2012 1:28 PM, "Stefan Priebe" <s.priebe@profihost.ag
>> <mailto:s.priebe@profihost.ag>> wrote:
>> >
>> > Am 30.12.2012 19:17, schrieb Samuel Just:
>> >>
>> >> This is somewhat more likely to have been a bug in the replication
>> logic
>> >> (there were a few fixed between 0.53 and 0.55). Had there been any
>> >> recent osd failures?
>> >
>> > Yes i was stressing CEPH with failures (power, link, disk, ...).
>> >
>> > Stefan
>> >
>> >> On Dec 24, 2012 10:55 PM, "Sage Weil" <sage@inktank.com
>> <mailto:sage@inktank.com>
>> >> <mailto:sage@inktank.com <mailto:sage@inktank.com>>> wrote:
>> >>
>> >> On Tue, 25 Dec 2012, Stefan Priebe wrote:
>> >> > Hello list,
>> >> >
>> >> > today i got the following ceph status output:
>> >> > 2012-12-25 02:57:00.632945 mon.0 [INF] pgmap v1394388: 7632
>> pgs: 7631
>> >> > active+clean, 1 active+clean+inconsistent; 151 GB data, 307 GB
>> >> used, 5028 GB /
>> >> > 5336 GB avail
>> >> >
>> >> >
>> >> > i then grepped the inconsistent pg by:
>> >> > # ceph pg dump - | grep inconsistent
>> >> > 3.ccf 10 0 0 0 41037824 155930
>> >> 155930
>> >> > active+clean+inconsistent 2012-12-25 01:51:35.318459
>> 6243'2107
>> >> > 6190'9847 [14,42] [14,42] 6243'2107 2012-12-25
>> >> 01:51:35.318436
>> >> > 6007'2074 2012-12-23 01:51:24.386366
>> >> >
>> >> > and initiated a repair:
>> >> > # ceph pg repair 3.ccf
>> >> > instructing pg 3.ccf on osd.14 to repair
>> >> >
>> >> > The log output then was:
>> >> > 2012-12-25 02:56:59.056382 osd.14 [ERR] 3.ccf osd.42 missing
>> >> > 1c602ccf/rbd_data.4904d6b8b4567.0000000000000b84/head//3
>> >> > 2012-12-25 02:56:59.056385 osd.14 [ERR] 3.ccf osd.42 missing
>> >> > ceb55ccf/rbd_data.48cc66b8b4567.0000000000001538/head//3
>> >> > 2012-12-25 02:56:59.097989 osd.14 [ERR] 3.ccf osd.42 missing
>> >> > dba6bccf/rbd_data.4797d6b8b4567.00000000000015ad/head//3
>> >> > 2012-12-25 02:56:59.097991 osd.14 [ERR] 3.ccf osd.42 missing
>> >> > a4deccf/rbd_data.45f956b8b4567.00000000000003d5/head//3
>> >> > 2012-12-25 02:56:59.098022 osd.14 [ERR] 3.ccf repair 4 missing,
>> 0
>> >> inconsistent
>> >> > objects
>> >> > 2012-12-25 02:56:59.098046 osd.14 [ERR] 3.ccf repair 4 errors,
>> 4
>> >> fixed
>> >> >
>> >> > Why doesn't ceph repair this automatically? Ho could this
>> happen
>> >> at all?
>> >>
>> >> We just made some fixes to repair in next (it was broken sometime
>> >> between
>> >> ~0.53 and 0.55). The latest next should repair it. In general
>> we don't
>> >> repair automatically lest we inadvertantly propagate bad data or
>> paper
>> >> over a bug.
>> >>
>> >> As for the original source of the missing objects... I'm not sure.
>> >> There
>> >> were some fixed races related to backfill that could lead to an
>> object
>> >> being missed, but Sam would know more about how likely that
>> actually is.
>> >>
>> >> sage
>> >> --
>> >> To unsubscribe from this list: send the line "unsubscribe
>> ceph-devel" in
>> >> the body of a message to majordomo@vger.kernel.org
>> <mailto:majordomo@vger.kernel.org>
>> >> <mailto:majordomo@vger.kernel.org
>>
>> <mailto:majordomo@vger.kernel.org>>
>> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
>> >>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: automatic repair of inconsistent pg?
2012-12-31 19:21 ` Samuel Just
@ 2013-01-01 20:12 ` Stefan Priebe
0 siblings, 0 replies; 7+ messages in thread
From: Stefan Priebe @ 2013-01-01 20:12 UTC (permalink / raw)
To: Samuel Just; +Cc: ceph-devel@vger.kernel.org, Sage Weil
OK thanks! Will change that.
Am 31.12.2012 20:21, schrieb Samuel Just:
> The ceph-osd relies on fs barriers for correctness. You will want to
> remove the nobarrier option to prevent future corruption.
> -Sam
>
> On Mon, Dec 31, 2012 at 3:59 AM, Stefan Priebe <s.priebe@profihost.ag> wrote:
>> Am 31.12.2012 02:10, schrieb Samuel Just:
>>
>>> Are you using xfs? If so, what mount options?
>>
>>
>> Yes,
>> noatime,nodiratime,nobarrier,logbufs=8,logbsize=256k
>>
>> Stefan
>>
>>>
>>> On Dec 30, 2012 1:28 PM, "Stefan Priebe" <s.priebe@profihost.ag
>>> <mailto:s.priebe@profihost.ag>> wrote:
>>> >
>>> > Am 30.12.2012 19:17, schrieb Samuel Just:
>>> >>
>>> >> This is somewhat more likely to have been a bug in the replication
>>> logic
>>> >> (there were a few fixed between 0.53 and 0.55). Had there been any
>>> >> recent osd failures?
>>> >
>>> > Yes i was stressing CEPH with failures (power, link, disk, ...).
>>> >
>>> > Stefan
>>> >
>>> >> On Dec 24, 2012 10:55 PM, "Sage Weil" <sage@inktank.com
>>> <mailto:sage@inktank.com>
>>> >> <mailto:sage@inktank.com <mailto:sage@inktank.com>>> wrote:
>>> >>
>>> >> On Tue, 25 Dec 2012, Stefan Priebe wrote:
>>> >> > Hello list,
>>> >> >
>>> >> > today i got the following ceph status output:
>>> >> > 2012-12-25 02:57:00.632945 mon.0 [INF] pgmap v1394388: 7632
>>> pgs: 7631
>>> >> > active+clean, 1 active+clean+inconsistent; 151 GB data, 307 GB
>>> >> used, 5028 GB /
>>> >> > 5336 GB avail
>>> >> >
>>> >> >
>>> >> > i then grepped the inconsistent pg by:
>>> >> > # ceph pg dump - | grep inconsistent
>>> >> > 3.ccf 10 0 0 0 41037824 155930
>>> >> 155930
>>> >> > active+clean+inconsistent 2012-12-25 01:51:35.318459
>>> 6243'2107
>>> >> > 6190'9847 [14,42] [14,42] 6243'2107 2012-12-25
>>> >> 01:51:35.318436
>>> >> > 6007'2074 2012-12-23 01:51:24.386366
>>> >> >
>>> >> > and initiated a repair:
>>> >> > # ceph pg repair 3.ccf
>>> >> > instructing pg 3.ccf on osd.14 to repair
>>> >> >
>>> >> > The log output then was:
>>> >> > 2012-12-25 02:56:59.056382 osd.14 [ERR] 3.ccf osd.42 missing
>>> >> > 1c602ccf/rbd_data.4904d6b8b4567.0000000000000b84/head//3
>>> >> > 2012-12-25 02:56:59.056385 osd.14 [ERR] 3.ccf osd.42 missing
>>> >> > ceb55ccf/rbd_data.48cc66b8b4567.0000000000001538/head//3
>>> >> > 2012-12-25 02:56:59.097989 osd.14 [ERR] 3.ccf osd.42 missing
>>> >> > dba6bccf/rbd_data.4797d6b8b4567.00000000000015ad/head//3
>>> >> > 2012-12-25 02:56:59.097991 osd.14 [ERR] 3.ccf osd.42 missing
>>> >> > a4deccf/rbd_data.45f956b8b4567.00000000000003d5/head//3
>>> >> > 2012-12-25 02:56:59.098022 osd.14 [ERR] 3.ccf repair 4 missing,
>>> 0
>>> >> inconsistent
>>> >> > objects
>>> >> > 2012-12-25 02:56:59.098046 osd.14 [ERR] 3.ccf repair 4 errors,
>>> 4
>>> >> fixed
>>> >> >
>>> >> > Why doesn't ceph repair this automatically? Ho could this
>>> happen
>>> >> at all?
>>> >>
>>> >> We just made some fixes to repair in next (it was broken sometime
>>> >> between
>>> >> ~0.53 and 0.55). The latest next should repair it. In general
>>> we don't
>>> >> repair automatically lest we inadvertantly propagate bad data or
>>> paper
>>> >> over a bug.
>>> >>
>>> >> As for the original source of the missing objects... I'm not sure.
>>> >> There
>>> >> were some fixed races related to backfill that could lead to an
>>> object
>>> >> being missed, but Sam would know more about how likely that
>>> actually is.
>>> >>
>>> >> sage
>>> >> --
>>> >> To unsubscribe from this list: send the line "unsubscribe
>>> ceph-devel" in
>>> >> the body of a message to majordomo@vger.kernel.org
>>> <mailto:majordomo@vger.kernel.org>
>>> >> <mailto:majordomo@vger.kernel.org
>>>
>>> <mailto:majordomo@vger.kernel.org>>
>>> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>> >>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2013-01-01 20:12 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-12-25 2:01 automatic repair of inconsistent pg? Stefan Priebe
2012-12-25 6:54 ` Sage Weil
2012-12-30 18:22 ` Samuel Just
[not found] ` <CA+4uBUbDVC0pKEfGfHmEuaVvZsoHWyZrxoE+vrFSddgWLGeELQ@mail.gmail.com>
2012-12-30 19:28 ` Stefan Priebe
[not found] ` <CA+4uBUb81Gv-4vjKTd8UvV7V8Ep7PG3roug30+0e9hHksTc35g@mail.gmail.com>
2012-12-31 11:59 ` Stefan Priebe
2012-12-31 19:21 ` Samuel Just
2013-01-01 20:12 ` Stefan Priebe
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.