[PATCH] Fix for corrupted ceph cluster

[PATCH] Fix for corrupted ceph cluster

[Daniel Poelzleithner]

Hi,

I wrote a small patch that ignores object_trim requests when he does not
find the context of this request.
We have a node that fails to start permanently and there is no way to
get all nodes back up.

As far as I understood, deleting something that does not exist should
not cause an assert. It is wired, but should not cause abort.

This is regarding bug http://tracker.ceph.com/issues/6101

Any help is highly appreciated.

kind regards
 Daniel


---
 src/osd/ReplicatedPG.cc | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/src/osd/ReplicatedPG.cc b/src/osd/ReplicatedPG.cc
index 90d3e1d..d7e0b62 100644
--- a/src/osd/ReplicatedPG.cc
+++ b/src/osd/ReplicatedPG.cc
@@ -1491,7 +1491,7 @@ ReplicatedPG::RepGather
*ReplicatedPG::trim_object(const hobject_t &coid)
   int r = find_object_context(coid, &obc, false, NULL);
   if (r == -ENOENT || coid.snap != obc->obs.oi.soid.snap) {
     derr << __func__ << "could not find coid " << coid << dendl;
-    assert(0);
+    return NULL;
   }
   assert(r == 0);
   assert(obc->registered);
@@ -7866,7 +7866,10 @@ boost::statechart::result
ReplicatedPG::TrimmingObjects::react(const SnapTrim&)

   dout(10) << "TrimmingObjects react trimming " << pos << dendl;
   RepGather *repop = pg->trim_object(pos);
-  assert(repop);
+  if (!repop) {
+      derr << "TrimmingObjects failed " << pos << dendl;
+      return discard_event();
+  }

   repop->queue_snap_trimmer = true;
   eversion_t old_last_update = pg->pg_log.get_head();
-- 
1.8.5.3

Re: [PATCH] Fix for corrupted ceph cluster

[Daniel Poelzleithner]

On 02/11/2014 07:48 AM, Daniel Poelzleithner wrote:

> I wrote a small patch that ignores object_trim requests when he does not
> find the context of this request.
> We have a node that fails to start permanently and there is no way to
> get all nodes back up.
[...]
> This is regarding bug http://tracker.ceph.com/issues/6101

The patch now ran for 2 weeks and the 4th node is working again.
I think this patch is safe to apply, but not fixing the underlying problem.
Some state in ceph causes the delete event to be triggered every some
seconds and causes a log entry to be generated.

Do you need more informations to find the cause ? This definitely is
some wired internal state and is no race condition.


kind regards
 Daniel

Re: [PATCH] Fix for corrupted ceph cluster

[Sage Weil]

On Thu, 6 Mar 2014, Daniel Poelzleithner wrote:
> On 02/11/2014 07:48 AM, Daniel Poelzleithner wrote:
> 
> > I wrote a small patch that ignores object_trim requests when he does not
> > find the context of this request.
> > We have a node that fails to start permanently and there is no way to
> > get all nodes back up.
> [...]
> > This is regarding bug http://tracker.ceph.com/issues/6101
> 
> The patch now ran for 2 weeks and the 4th node is working again.
> I think this patch is safe to apply, but not fixing the underlying problem.
> Some state in ceph causes the delete event to be triggered every some
> seconds and causes a log entry to be generated.
> 
> Do you need more informations to find the cause ? This definitely is
> some wired internal state and is no race condition.

Can you try, instead of the discard_event, to do

    post_event(SnapTrim());
    return transit< WaitingOnReplicas >();

and see if that lets it move past the bad entry?

Thanks-
sage