From: Stefan Priebe <s.priebe@profihost.ag>
To: Samuel Just <sam.just@inktank.com>
Cc: "ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>,
Sage Weil <sage@inktank.com>
Subject: Re: automatic repair of inconsistent pg?
Date: Mon, 31 Dec 2012 12:59:07 +0100 [thread overview]
Message-ID: <50E17E0B.805@profihost.ag> (raw)
In-Reply-To: <CA+4uBUb81Gv-4vjKTd8UvV7V8Ep7PG3roug30+0e9hHksTc35g@mail.gmail.com>
Am 31.12.2012 02:10, schrieb Samuel Just:
> Are you using xfs? If so, what mount options?
Yes,
noatime,nodiratime,nobarrier,logbufs=8,logbsize=256k
Stefan
>
> On Dec 30, 2012 1:28 PM, "Stefan Priebe" <s.priebe@profihost.ag
> <mailto:s.priebe@profihost.ag>> wrote:
> >
> > Am 30.12.2012 19:17, schrieb Samuel Just:
> >>
> >> This is somewhat more likely to have been a bug in the replication logic
> >> (there were a few fixed between 0.53 and 0.55). Had there been any
> >> recent osd failures?
> >
> > Yes i was stressing CEPH with failures (power, link, disk, ...).
> >
> > Stefan
> >
> >> On Dec 24, 2012 10:55 PM, "Sage Weil" <sage@inktank.com
> <mailto:sage@inktank.com>
> >> <mailto:sage@inktank.com <mailto:sage@inktank.com>>> wrote:
> >>
> >> On Tue, 25 Dec 2012, Stefan Priebe wrote:
> >> > Hello list,
> >> >
> >> > today i got the following ceph status output:
> >> > 2012-12-25 02:57:00.632945 mon.0 [INF] pgmap v1394388: 7632
> pgs: 7631
> >> > active+clean, 1 active+clean+inconsistent; 151 GB data, 307 GB
> >> used, 5028 GB /
> >> > 5336 GB avail
> >> >
> >> >
> >> > i then grepped the inconsistent pg by:
> >> > # ceph pg dump - | grep inconsistent
> >> > 3.ccf 10 0 0 0 41037824 155930
> >> 155930
> >> > active+clean+inconsistent 2012-12-25 01:51:35.318459
> 6243'2107
> >> > 6190'9847 [14,42] [14,42] 6243'2107 2012-12-25
> >> 01:51:35.318436
> >> > 6007'2074 2012-12-23 01:51:24.386366
> >> >
> >> > and initiated a repair:
> >> > # ceph pg repair 3.ccf
> >> > instructing pg 3.ccf on osd.14 to repair
> >> >
> >> > The log output then was:
> >> > 2012-12-25 02:56:59.056382 osd.14 [ERR] 3.ccf osd.42 missing
> >> > 1c602ccf/rbd_data.4904d6b8b4567.0000000000000b84/head//3
> >> > 2012-12-25 02:56:59.056385 osd.14 [ERR] 3.ccf osd.42 missing
> >> > ceb55ccf/rbd_data.48cc66b8b4567.0000000000001538/head//3
> >> > 2012-12-25 02:56:59.097989 osd.14 [ERR] 3.ccf osd.42 missing
> >> > dba6bccf/rbd_data.4797d6b8b4567.00000000000015ad/head//3
> >> > 2012-12-25 02:56:59.097991 osd.14 [ERR] 3.ccf osd.42 missing
> >> > a4deccf/rbd_data.45f956b8b4567.00000000000003d5/head//3
> >> > 2012-12-25 02:56:59.098022 osd.14 [ERR] 3.ccf repair 4 missing, 0
> >> inconsistent
> >> > objects
> >> > 2012-12-25 02:56:59.098046 osd.14 [ERR] 3.ccf repair 4 errors, 4
> >> fixed
> >> >
> >> > Why doesn't ceph repair this automatically? Ho could this happen
> >> at all?
> >>
> >> We just made some fixes to repair in next (it was broken sometime
> >> between
> >> ~0.53 and 0.55). The latest next should repair it. In general
> we don't
> >> repair automatically lest we inadvertantly propagate bad data or
> paper
> >> over a bug.
> >>
> >> As for the original source of the missing objects... I'm not sure.
> >> There
> >> were some fixed races related to backfill that could lead to an
> object
> >> being missed, but Sam would know more about how likely that
> actually is.
> >>
> >> sage
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe
> ceph-devel" in
> >> the body of a message to majordomo@vger.kernel.org
> <mailto:majordomo@vger.kernel.org>
> >> <mailto:majordomo@vger.kernel.org
> <mailto:majordomo@vger.kernel.org>>
> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >>
next prev parent reply other threads:[~2012-12-31 11:59 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-12-25 2:01 automatic repair of inconsistent pg? Stefan Priebe
2012-12-25 6:54 ` Sage Weil
2012-12-30 18:22 ` Samuel Just
[not found] ` <CA+4uBUbDVC0pKEfGfHmEuaVvZsoHWyZrxoE+vrFSddgWLGeELQ@mail.gmail.com>
2012-12-30 19:28 ` Stefan Priebe
[not found] ` <CA+4uBUb81Gv-4vjKTd8UvV7V8Ep7PG3roug30+0e9hHksTc35g@mail.gmail.com>
2012-12-31 11:59 ` Stefan Priebe [this message]
2012-12-31 19:21 ` Samuel Just
2013-01-01 20:12 ` Stefan Priebe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=50E17E0B.805@profihost.ag \
--to=s.priebe@profihost.ag \
--cc=ceph-devel@vger.kernel.org \
--cc=sage@inktank.com \
--cc=sam.just@inktank.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox