From: Olivier Bonvalet <ceph.list-PaEMFeTk6C1QFI55V6+gNQ@public.gmane.org>
To: ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Cc: "ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org"
<ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org>
Subject: Re: Scrub shutdown the OSD process
Date: Sat, 20 Apr 2013 09:10:39 +0200 [thread overview]
Message-ID: <1366441839.13398.6.camel@localhost> (raw)
In-Reply-To: <1366224725.14655.30.camel@localhost>
Le mercredi 17 avril 2013 à 20:52 +0200, Olivier Bonvalet a écrit :
> What I didn't understand is why the OSD process crash, instead of
> marking that PG "corrupted", and does that PG really "corrupted" are
> is
> this just an OSD bug ?
Once again, a bit more informations : by searching informations about
one of this faulty PG (3.d), I found that :
-592> 2013-04-20 08:31:56.838280 7f0f41d1b700 0 log [ERR] : 3.d osd.25 inconsistent snapcolls on a8620b0d/rb.0.15c26.238e1f29.000000004603/12d7//3 found expected 12d7
-591> 2013-04-20 08:31:56.838284 7f0f41d1b700 0 log [ERR] : 3.d osd.4 inconsistent snapcolls on a8620b0d/rb.0.15c26.238e1f29.000000004603/12d7//3 found expected 12d7
-590> 2013-04-20 08:31:56.838290 7f0f41d1b700 0 log [ERR] : 3.d osd.4: soid a8620b0d/rb.0.15c26.238e1f29.000000004603/12d7//3 size 4194304 != known size 0
-589> 2013-04-20 08:31:56.838292 7f0f41d1b700 0 log [ERR] : 3.d osd.11 inconsistent snapcolls on a8620b0d/rb.0.15c26.238e1f29.000000004603/12d7//3 found expected 12d7
-588> 2013-04-20 08:31:56.838294 7f0f41d1b700 0 log [ERR] : 3.d osd.11: soid a8620b0d/rb.0.15c26.238e1f29.000000004603/12d7//3 size 4194304 != known size 0
-587> 2013-04-20 08:31:56.838395 7f0f41d1b700 0 log [ERR] : scrub 3.d a8620b0d/rb.0.15c26.238e1f29.000000004603/12d7//3 on disk size (0) does not match object info size (4194304)
I prefered to verify, so I found that :
# md5sum /var/lib/ceph/osd/ceph-*/current/3.d_head/DIR_D/DIR_0/DIR_B/DIR_0/rb.0.15c26.238e1f29.000000004603__12d7_A8620B0D__3
217ac2518dfe9e1502e5bfedb8be29b8 /var/lib/ceph/osd/ceph-4/current/3.d_head/DIR_D/DIR_0/DIR_B/DIR_0/rb.0.15c26.238e1f29.000000004603__12d7_A8620B0D__3 (4MB)
217ac2518dfe9e1502e5bfedb8be29b8 /var/lib/ceph/osd/ceph-11/current/3.d_head/DIR_D/DIR_0/DIR_B/DIR_0/rb.0.15c26.238e1f29.000000004603__12d7_A8620B0D__3 (4MB)
d41d8cd98f00b204e9800998ecf8427e /var/lib/ceph/osd/ceph-25/current/3.d_head/DIR_D/DIR_0/DIR_B/DIR_0/rb.0.15c26.238e1f29.000000004603__12d7_A8620B0D__3 (0B)
So this object is identical on OSD 4 and 11, but is empty on OSD 25.
Since 4 is the master, this should not be a problem, so I try a repair,
without any success :
ceph pg repair 3.d
Is there a way to force rewrite of this replica ?
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
next prev parent reply other threads:[~2013-04-20 7:10 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <1366018923.3018.3.camel@localhost>
[not found] ` <CAPYLRziFNxnUhyxHRUCzZBF6oQwrWt8YdxBtDrKzMMNuyuW=YQ@mail.gmail.com>
[not found] ` <1366046351.26980.0.camel@localhost>
2013-04-15 17:57 ` [ceph-users] Scrub shutdown the OSD process Gregory Farnum
[not found] ` <CAPYLRzjz3FVvm5OH-TBN3R8tmv8LREFYKTEX41Zk7DTLjoPXkA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-04-15 18:32 ` Olivier Bonvalet
2013-04-16 6:56 ` Olivier Bonvalet
2013-04-17 9:48 ` [ceph-users] " Olivier Bonvalet
2013-04-17 18:52 ` Olivier Bonvalet
2013-04-20 7:10 ` Olivier Bonvalet [this message]
2013-04-22 17:05 ` Scrub shutdown the OSD process / data loss Olivier Bonvalet
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1366441839.13398.6.camel@localhost \
--to=ceph.list-paemfetk6c1qfi55v6+gnq@public.gmane.org \
--cc=ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
--cc=ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.