From mboxrd@z Thu Jan 1 00:00:00 1970 From: Greg Farnum Subject: Re: pgs stuck inactive Date: Mon, 16 Apr 2012 17:06:42 -0700 Message-ID: <90B1EE3CB1C7494B9B097AD5E4BCF272@dreamhost.com> References: <3154FE08568349E9B9EB0EBBCE4EDFB2@dreamhost.com> <8260F00ADA384FEDBA6AA9873273BB38@dreamhost.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Return-path: Received: from mail-ob0-f174.google.com ([209.85.214.174]:58360 "EHLO mail-ob0-f174.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754755Ab2DQAGq convert rfc822-to-8bit (ORCPT ); Mon, 16 Apr 2012 20:06:46 -0400 Received: by obbta14 with SMTP id ta14so4331239obb.19 for ; Mon, 16 Apr 2012 17:06:45 -0700 (PDT) In-Reply-To: Content-Disposition: inline Sender: ceph-devel-owner@vger.kernel.org List-ID: To: Damien Churchill Cc: Samuel Just , ceph-devel@vger.kernel.org On Friday, April 13, 2012 at 12:42 PM, Damien Churchill wrote: > Hi, > =20 > On 13 April 2012 20:30, Greg Farnum wrote: > > On Thursday, April 12, 2012 at 8:29 AM, Damien Churchill wrote: > > > On 11 April 2012 00:40, Greg Farnum wrote: > > > > =20 > > > > A quick glance through these shows that all the pg_temp request= s aren't actually requesting any changes from the monitor. It's either = a very serious mon bug which happened a while ago (unlikely, given the = restarts and ongoing map changes, etc), or an OSD bug. I think we want = logs from both osd.0 and osd.3 at the same time, from what I'm seeing. = :) > > > > -Greg > > > =20 > > > =20 > > > =20 > > > =20 > > > =20 > > > Just to make sure all bases are covered: > > > =20 > > > http://damoxc.net/ceph/ceph-logs-20120412142537.tar.gz > > > =20 > > > This contains all 5 osd logs and all 3 monitor logs, everything > > > restarted with debug logging prior to capturing the logs. > > =20 > > =20 > > =20 > > I (and Sam) spent some time looking at this very closely. It contin= ues to tell me that the OSD and the monitor are disagreeing on whether = osd 3 should be in the pg temp set for some things, but they seem to ag= ree on everything else=E2=80=A6. > > Can you zip up for me: > > 1) The files matching osdmap* of osd0's store from the current/meta= / directory, > > 2) The contents of your lead monitor's osdmap and osdmap_full direc= tories? > =20 > =20 > =20 > Here they are > =20 > http://damoxc.net/ceph/osdmap.0.tar.gz > http://damoxc.net/ceph/mon.node21.osdmap.tar.gz > =20 > Hopefully I got the right files :) Yep! We looked into this more today and have discovered some definite oddnes= s. Have you by any chance tried to change the number of PGs in your poo= ls? -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" i= n the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html