From mboxrd@z Thu Jan 1 00:00:00 1970 From: Greg Farnum Subject: Re: pgs stuck inactive Date: Wed, 18 Apr 2012 11:41:37 -0700 Message-ID: References: <3154FE08568349E9B9EB0EBBCE4EDFB2@dreamhost.com> <8260F00ADA384FEDBA6AA9873273BB38@dreamhost.com> <90B1EE3CB1C7494B9B097AD5E4BCF272@dreamhost.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Return-path: Received: from mail-yx0-f174.google.com ([209.85.213.174]:57552 "EHLO mail-yx0-f174.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752938Ab2DRSln (ORCPT ); Wed, 18 Apr 2012 14:41:43 -0400 Received: by yenl12 with SMTP id l12so3897002yen.19 for ; Wed, 18 Apr 2012 11:41:42 -0700 (PDT) In-Reply-To: Content-Disposition: inline Sender: ceph-devel-owner@vger.kernel.org List-ID: To: Damien Churchill Cc: ceph-devel@vger.kernel.org On Tuesday, April 17, 2012 at 11:41 PM, Damien Churchill wrote: > On 17 April 2012 17:49, Greg Farnum wrote: > > Do you know what version this was created with, and what upgrades you've been through? My best guess right now is that there's a problem with the encoding and decoding that I'm going to have to track down, and more context will make it a lot easier. :) > > > > Hmmm that's testing my memory, I'd say that cluster has been alive at > least since 0.34. Occasionally I think there was a version skipped, > not sure if that could cause any issues? Okay. So the good news is that we can see what's broken now and have a kludge to prevent it happening to others; the bad news is we still have no idea how it actually occurred. :( But I don't think it's worth investing the time given what we have available, so all we can do is repair your cluster. Are you building your binaries from source, and can you run a patched version of the monitors? If you can I'll give you a patch to enable a simple command that should make things work; otherwise we'll need to start editing things by hand. (Yucky) -Greg