cuttlefish countdown

cuttlefish countdown

[Sage Weil]

Hi everyone-

We are down to a handful of urgent bugs (3!) and a cuttlefish release date 
that is less than a week away.  Thank you to everyone who has been 
involved in coding, testing, and stabilizing this release.  We are close!

If you would like to test the current release candidate, your efforts 
would be much appreciated!  For deb systems, you can do

 wget -q -O- 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc' | sudo apt-key add -
 echo deb http://gitbuilder.ceph.com/ceph-deb-$(lsb_release -sc)-x86_64-basic/ref/next $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list

For rpm users you can find packages at

 http://gitbuilder.ceph.com/ceph-rpm-centos6-x86_64-basic/ref/next/
 http://gitbuilder.ceph.com/ceph-rpm-fc17-x86_64-basic/ref/next/
 http://gitbuilder.ceph.com/ceph-rpm-fc18-x86_64-basic/ref/next/

A draft of the release notes is up at

 http://ceph.com/docs/master/release-notes/#v0-61

Let me know if I've missed anything!

sage

Re: cuttlefish countdown

[Wido den Hollander]

Hi,

On 04/25/2013 01:46 AM, Sage Weil wrote:
> Hi everyone-
>
> We are down to a handful of urgent bugs (3!) and a cuttlefish release date
> that is less than a week away.  Thank you to everyone who has been
> involved in coding, testing, and stabilizing this release.  We are close!
>

I'm trying to find the 3 urgent bugs for Cuttlefish, but I can't seem to 
fetch them out of Redmine: 
http://tracker.ceph.com/projects/1/issues?sort=position

That doesn't show me any open urgent bugs? Trying to figure out what's 
still open.

Wido

> If you would like to test the current release candidate, your efforts
> would be much appreciated!  For deb systems, you can do
>
>   wget -q -O- 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc' | sudo apt-key add -
>   echo deb http://gitbuilder.ceph.com/ceph-deb-$(lsb_release -sc)-x86_64-basic/ref/next $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
>
> For rpm users you can find packages at
>
>   http://gitbuilder.ceph.com/ceph-rpm-centos6-x86_64-basic/ref/next/
>   http://gitbuilder.ceph.com/ceph-rpm-fc17-x86_64-basic/ref/next/
>   http://gitbuilder.ceph.com/ceph-rpm-fc18-x86_64-basic/ref/next/
>
> A draft of the release notes is up at
>
>   http://ceph.com/docs/master/release-notes/#v0-61
>
> Let me know if I've missed anything!
>
> sage
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>


-- 
Wido den Hollander
42on B.V.

Phone: +31 (0)20 700 9902
Skype: contact42on

Re: cuttlefish countdown

[Kevin Decherf]

Hi,

On Thu, Apr 25, 2013 at 10:17:47AM +0200, Wido den Hollander wrote:
> Hi,
> 
> On 04/25/2013 01:46 AM, Sage Weil wrote:
> >Hi everyone-
> >
> >We are down to a handful of urgent bugs (3!) and a cuttlefish release date
> >that is less than a week away.  Thank you to everyone who has been
> >involved in coding, testing, and stabilizing this release.  We are close!
> >
> 
> I'm trying to find the 3 urgent bugs for Cuttlefish, but I can't
> seem to fetch them out of Redmine:
> http://tracker.ceph.com/projects/1/issues?sort=position
> 
> That doesn't show me any open urgent bugs? Trying to figure out
> what's still open.

http://tracker.ceph.com/projects/ceph/issues?utf8=%E2%9C%93&set_filter=1&f[]=status_id&op[status_id]=o&f[]=fixed_version_id&op[fixed_version_id]=%3D&v[fixed_version_id][]=138&f[]=&c[]=project&c[]=tracker&c[]=status&c[]=priority&c[]=subject&c[]=assigned_to&c[]=updated_on&c[]=category&c[]=fixed_version&c[]=story_points&c[]=cf_3&c[]=cf_4&group_by=

That's what you are looking for?

-- 
Kevin Decherf - @Kdecherf
GPG C610 FE73 E706 F968 612B E4B2 108A BD75 A81E 6E2F
http://kdecherf.com

Re: cuttlefish countdown

[Ian Colle]

On 4/25/13 4:03 AM, "Kevin Decherf" <kevin@kdecherf.com> wrote:

>Hi,
>
>On Thu, Apr 25, 2013 at 10:17:47AM +0200, Wido den Hollander wrote:
>> Hi,
>> 
>> On 04/25/2013 01:46 AM, Sage Weil wrote:
>> >Hi everyone-
>> >
>> >We are down to a handful of urgent bugs (3!) and a cuttlefish release
>>date
>> >that is less than a week away.  Thank you to everyone who has been
>> >involved in coding, testing, and stabilizing this release.  We are
>>close!
>> >
>> 
>> I'm trying to find the 3 urgent bugs for Cuttlefish, but I can't
>> seem to fetch them out of Redmine:
>> http://tracker.ceph.com/projects/1/issues?sort=position
>> 
>> That doesn't show me any open urgent bugs? Trying to figure out
>> what's still open.
>
>http://tracker.ceph.com/projects/ceph/issues?utf8=%E2%9C%93&set_filter=1&f
>[]=status_id&op[status_id]=o&f[]=fixed_version_id&op[fixed_version_id]=%3D
>&v[fixed_version_id][]=138&f[]=&c[]=project&c[]=tracker&c[]=status&c[]=pri
>ority&c[]=subject&c[]=assigned_to&c[]=updated_on&c[]=category&c[]=fixed_ve
>rsion&c[]=story_points&c[]=cf_3&c[]=cf_4&group_by=
>
>That's what you are looking for?

Try the Bug Queue filter:
http://tracker.ceph.com/projects/ceph/issues?query_id=27


>
>-- 
>Kevin Decherf - @Kdecherf
>GPG C610 FE73 E706 F968 612B E4B2 108A BD75 A81E 6E2F
>http://kdecherf.com
>--
>To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: cuttlefish countdown

[Sage Weil]

http://tracker.ceph.com/projects/ceph/issues?query_id=27

It's the 'Bug queue' link on the lower right under 'custom queries'.  Now 
there are 6 (ignoring pending backport)... :(

s

On Thu, 25 Apr 2013, Wido den Hollander wrote:

> Hi,
> 
> On 04/25/2013 01:46 AM, Sage Weil wrote:
> > Hi everyone-
> > 
> > We are down to a handful of urgent bugs (3!) and a cuttlefish release date
> > that is less than a week away.  Thank you to everyone who has been
> > involved in coding, testing, and stabilizing this release.  We are close!
> > 
> 
> I'm trying to find the 3 urgent bugs for Cuttlefish, but I can't seem to fetch
> them out of Redmine: http://tracker.ceph.com/projects/1/issues?sort=position
> 
> That doesn't show me any open urgent bugs? Trying to figure out what's still
> open.
> 
> Wido
> 
> > If you would like to test the current release candidate, your efforts
> > would be much appreciated!  For deb systems, you can do
> > 
> >   wget -q -O-
> > 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc' | sudo
> > apt-key add -
> >   echo deb http://gitbuilder.ceph.com/ceph-deb-$(lsb_release
> > -sc)-x86_64-basic/ref/next $(lsb_release -sc) main | sudo tee
> > /etc/apt/sources.list.d/ceph.list
> > 
> > For rpm users you can find packages at
> > 
> >   http://gitbuilder.ceph.com/ceph-rpm-centos6-x86_64-basic/ref/next/
> >   http://gitbuilder.ceph.com/ceph-rpm-fc17-x86_64-basic/ref/next/
> >   http://gitbuilder.ceph.com/ceph-rpm-fc18-x86_64-basic/ref/next/
> > 
> > A draft of the release notes is up at
> > 
> >   http://ceph.com/docs/master/release-notes/#v0-61
> > 
> > Let me know if I've missed anything!
> > 
> > sage
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 
> 
> -- 
> Wido den Hollander
> 42on B.V.
> 
> Phone: +31 (0)20 700 9902
> Skype: contact42on
> 
> 

Re: cuttlefish countdown -- OSD doesn't get marked out

[Martin Mailand]

Hi,

if I shutdown an OSD, the OSD gets marked down after 20 seconds, after
300 seconds the osd should get marked out, an the cluster should resync.
But that doesn't happened, the OSD stays in the status down/in forever,
therefore the cluster stays forever degraded.
I can reproduce it with a new installed cluster.

If I manually set the osd out (ceph osd out 1), the cluster resync
starts immediately.

I think thats a release critical bug, because the cluster health is not
automatically recovered.

And I reported this behavior a while ago
http://article.gmane.org/gmane.comp.file-systems.ceph.user/603/

-martin


Log:


root@store1:~# ceph -s
   health HEALTH_OK
   monmap e1: 3 mons at
{a=192.168.195.31:6789/0,b=192.168.195.33:6789/0,c=192.168.195.35:6789/0},
election epoch 82, quorum 0,1,2 a,b,c
   osdmap e204: 24 osds: 24 up, 24 in
    pgmap v106709: 5056 pgs: 5056 active+clean; 526 GB data, 1068 GB
used, 173 TB / 174 TB avail
   mdsmap e1: 0/0/1 up

root@store1:~# ceph --version
ceph version 0.60 (f26f7a39021dbf440c28d6375222e21c94fe8e5c)
root@store1:~# /etc/init.d/ceph stop osd.1
=== osd.1 ===
Stopping Ceph osd.1 on store1...bash: warning: setlocale: LC_ALL: cannot
change locale (en_GB.utf8)
kill 5492...done
root@store1:~# ceph -s
   health HEALTH_OK
   monmap e1: 3 mons at
{a=192.168.195.31:6789/0,b=192.168.195.33:6789/0,c=192.168.195.35:6789/0},
election epoch 82, quorum 0,1,2 a,b,c
   osdmap e204: 24 osds: 24 up, 24 in
    pgmap v106709: 5056 pgs: 5056 active+clean; 526 GB data, 1068 GB
used, 173 TB / 174 TB avail
   mdsmap e1: 0/0/1 up

root@store1:~# date -R
Thu, 25 Apr 2013 13:09:54 +0200



root@store1:~# ceph -s && date -R
   health HEALTH_WARN 423 pgs degraded; 423 pgs stuck unclean; recovery
10999/269486 degraded (4.081%); 1/24 in osds are down
   monmap e1: 3 mons at
{a=192.168.195.31:6789/0,b=192.168.195.33:6789/0,c=192.168.195.35:6789/0},
election epoch 82, quorum 0,1,2 a,b,c
   osdmap e206: 24 osds: 23 up, 24 in
    pgmap v106715: 5056 pgs: 4633 active+clean, 423 active+degraded; 526
GB data, 1068 GB used, 173 TB / 174 TB avail; 10999/269486 degraded (4.081%)
   mdsmap e1: 0/0/1 up

Thu, 25 Apr 2013 13:10:14 +0200


root@store1:~# ceph -s && date -R
   health HEALTH_WARN 423 pgs degraded; 423 pgs stuck unclean; recovery
10999/269486 degraded (4.081%); 1/24 in osds are down
   monmap e1: 3 mons at
{a=192.168.195.31:6789/0,b=192.168.195.33:6789/0,c=192.168.195.35:6789/0},
election epoch 82, quorum 0,1,2 a,b,c
   osdmap e206: 24 osds: 23 up, 24 in
    pgmap v106719: 5056 pgs: 4633 active+clean, 423 active+degraded; 526
GB data, 1068 GB used, 173 TB / 174 TB avail; 10999/269486 degraded (4.081%)
   mdsmap e1: 0/0/1 up

Thu, 25 Apr 2013 13:23:01 +0200

On 25.04.2013 01:46, Sage Weil wrote:
> Hi everyone-
> 
> We are down to a handful of urgent bugs (3!) and a cuttlefish release date 
> that is less than a week away.  Thank you to everyone who has been 
> involved in coding, testing, and stabilizing this release.  We are close!
> 
> If you would like to test the current release candidate, your efforts 
> would be much appreciated!  For deb systems, you can do
> 
>  wget -q -O- 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc' | sudo apt-key add -
>  echo deb http://gitbuilder.ceph.com/ceph-deb-$(lsb_release -sc)-x86_64-basic/ref/next $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
> 
> For rpm users you can find packages at
> 
>  http://gitbuilder.ceph.com/ceph-rpm-centos6-x86_64-basic/ref/next/
>  http://gitbuilder.ceph.com/ceph-rpm-fc17-x86_64-basic/ref/next/
>  http://gitbuilder.ceph.com/ceph-rpm-fc18-x86_64-basic/ref/next/
> 
> A draft of the release notes is up at
> 
>  http://ceph.com/docs/master/release-notes/#v0-61
> 
> Let me know if I've missed anything!
> 
> sage
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: cuttlefish countdown -- OSD doesn't get marked out

[Wido den Hollander]

On 04/25/2013 02:07 PM, Martin Mailand wrote:
> Hi,
>
> if I shutdown an OSD, the OSD gets marked down after 20 seconds, after
> 300 seconds the osd should get marked out, an the cluster should resync.
> But that doesn't happened, the OSD stays in the status down/in forever,
> therefore the cluster stays forever degraded.
> I can reproduce it with a new installed cluster.
>
> If I manually set the osd out (ceph osd out 1), the cluster resync
> starts immediately.
>

Could you dump your osdmap? The first 10 lines would be interesting. 
There is a flag where you say "noosdout", could it be that the flag is set?

Wido

> I think thats a release critical bug, because the cluster health is not
> automatically recovered.
>
> And I reported this behavior a while ago
> http://article.gmane.org/gmane.comp.file-systems.ceph.user/603/
>
> -martin
>
>
> Log:
>
>
> root@store1:~# ceph -s
>     health HEALTH_OK
>     monmap e1: 3 mons at
> {a=192.168.195.31:6789/0,b=192.168.195.33:6789/0,c=192.168.195.35:6789/0},
> election epoch 82, quorum 0,1,2 a,b,c
>     osdmap e204: 24 osds: 24 up, 24 in
>      pgmap v106709: 5056 pgs: 5056 active+clean; 526 GB data, 1068 GB
> used, 173 TB / 174 TB avail
>     mdsmap e1: 0/0/1 up
>
> root@store1:~# ceph --version
> ceph version 0.60 (f26f7a39021dbf440c28d6375222e21c94fe8e5c)
> root@store1:~# /etc/init.d/ceph stop osd.1
> === osd.1 ===
> Stopping Ceph osd.1 on store1...bash: warning: setlocale: LC_ALL: cannot
> change locale (en_GB.utf8)
> kill 5492...done
> root@store1:~# ceph -s
>     health HEALTH_OK
>     monmap e1: 3 mons at
> {a=192.168.195.31:6789/0,b=192.168.195.33:6789/0,c=192.168.195.35:6789/0},
> election epoch 82, quorum 0,1,2 a,b,c
>     osdmap e204: 24 osds: 24 up, 24 in
>      pgmap v106709: 5056 pgs: 5056 active+clean; 526 GB data, 1068 GB
> used, 173 TB / 174 TB avail
>     mdsmap e1: 0/0/1 up
>
> root@store1:~# date -R
> Thu, 25 Apr 2013 13:09:54 +0200
>
>
>
> root@store1:~# ceph -s && date -R
>     health HEALTH_WARN 423 pgs degraded; 423 pgs stuck unclean; recovery
> 10999/269486 degraded (4.081%); 1/24 in osds are down
>     monmap e1: 3 mons at
> {a=192.168.195.31:6789/0,b=192.168.195.33:6789/0,c=192.168.195.35:6789/0},
> election epoch 82, quorum 0,1,2 a,b,c
>     osdmap e206: 24 osds: 23 up, 24 in
>      pgmap v106715: 5056 pgs: 4633 active+clean, 423 active+degraded; 526
> GB data, 1068 GB used, 173 TB / 174 TB avail; 10999/269486 degraded (4.081%)
>     mdsmap e1: 0/0/1 up
>
> Thu, 25 Apr 2013 13:10:14 +0200
>
>
> root@store1:~# ceph -s && date -R
>     health HEALTH_WARN 423 pgs degraded; 423 pgs stuck unclean; recovery
> 10999/269486 degraded (4.081%); 1/24 in osds are down
>     monmap e1: 3 mons at
> {a=192.168.195.31:6789/0,b=192.168.195.33:6789/0,c=192.168.195.35:6789/0},
> election epoch 82, quorum 0,1,2 a,b,c
>     osdmap e206: 24 osds: 23 up, 24 in
>      pgmap v106719: 5056 pgs: 4633 active+clean, 423 active+degraded; 526
> GB data, 1068 GB used, 173 TB / 174 TB avail; 10999/269486 degraded (4.081%)
>     mdsmap e1: 0/0/1 up
>
> Thu, 25 Apr 2013 13:23:01 +0200
>
> On 25.04.2013 01:46, Sage Weil wrote:
>> Hi everyone-
>>
>> We are down to a handful of urgent bugs (3!) and a cuttlefish release date
>> that is less than a week away.  Thank you to everyone who has been
>> involved in coding, testing, and stabilizing this release.  We are close!
>>
>> If you would like to test the current release candidate, your efforts
>> would be much appreciated!  For deb systems, you can do
>>
>>   wget -q -O- 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc' | sudo apt-key add -
>>   echo deb http://gitbuilder.ceph.com/ceph-deb-$(lsb_release -sc)-x86_64-basic/ref/next $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
>>
>> For rpm users you can find packages at
>>
>>   http://gitbuilder.ceph.com/ceph-rpm-centos6-x86_64-basic/ref/next/
>>   http://gitbuilder.ceph.com/ceph-rpm-fc17-x86_64-basic/ref/next/
>>   http://gitbuilder.ceph.com/ceph-rpm-fc18-x86_64-basic/ref/next/
>>
>> A draft of the release notes is up at
>>
>>   http://ceph.com/docs/master/release-notes/#v0-61
>>
>> Let me know if I've missed anything!
>>
>> sage
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>


-- 
Wido den Hollander
42on B.V.

Phone: +31 (0)20 700 9902
Skype: contact42on

Re: cuttlefish countdown -- OSD doesn't get marked out

[Martin Mailand]

Hi Wido,

I did not set the noosdout flag.

-martin

On 25.04.2013 14:56, Wido den Hollander wrote:
> Could you dump your osdmap? The first 10 lines would be interesting.
> There is a flag where you say "noosdout", could it be that the flag is set?
> 
> Wido


epoch 206
fsid 13538f8a-a9b5-4f57-ad72-b0f9bec56a3e
created 2013-03-28 20:19:20.790878
modified 2013-04-25 13:09:55.825509
flags

pool 0 'data' rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins
pg_num 1600 pgp_num 1600 last_change 1 owner 0 crash_replay_interval 45
pool 1 'metadata' rep size 2 min_size 1 crush_ruleset 1 object_hash
rjenkins pg_num 1600 pgp_num 1600 last_change 1 owner 0
pool 2 'rbd' rep size 2 min_size 1 crush_ruleset 2 object_hash rjenkins
pg_num 1600 pgp_num 1600 last_change 77 owner 0
        removed_snaps [1~1]
pool 3 'volumes' rep size 2 min_size 1 crush_ruleset 0 object_hash
rjenkins pg_num 128 pgp_num 128 last_change 199 owner 0
pool 4 'images' rep size 2 min_size 1 crush_ruleset 0 object_hash
rjenkins pg_num 128 pgp_num 128 last_change 204 owner 0
        removed_snaps [1~1]

max_osd 24

Re: cuttlefish countdown -- OSD doesn't get marked out

[Sage Weil]

On Thu, 25 Apr 2013, Martin Mailand wrote:
> Hi,
> 
> if I shutdown an OSD, the OSD gets marked down after 20 seconds, after
> 300 seconds the osd should get marked out, an the cluster should resync.
> But that doesn't happened, the OSD stays in the status down/in forever,
> therefore the cluster stays forever degraded.
> I can reproduce it with a new installed cluster.
> 
> If I manually set the osd out (ceph osd out 1), the cluster resync
> starts immediately.
> 
> I think thats a release critical bug, because the cluster health is not
> automatically recovered.

What is the output from 'ceph osd tree' and the contents of your 
[mon*] sections of ceph.conf?

Thanks!
sage


> 
> And I reported this behavior a while ago
> http://article.gmane.org/gmane.comp.file-systems.ceph.user/603/
> 
> -martin
> 
> 
> Log:
> 
> 
> root@store1:~# ceph -s
>    health HEALTH_OK
>    monmap e1: 3 mons at
> {a=192.168.195.31:6789/0,b=192.168.195.33:6789/0,c=192.168.195.35:6789/0},
> election epoch 82, quorum 0,1,2 a,b,c
>    osdmap e204: 24 osds: 24 up, 24 in
>     pgmap v106709: 5056 pgs: 5056 active+clean; 526 GB data, 1068 GB
> used, 173 TB / 174 TB avail
>    mdsmap e1: 0/0/1 up
> 
> root@store1:~# ceph --version
> ceph version 0.60 (f26f7a39021dbf440c28d6375222e21c94fe8e5c)
> root@store1:~# /etc/init.d/ceph stop osd.1
> === osd.1 ===
> Stopping Ceph osd.1 on store1...bash: warning: setlocale: LC_ALL: cannot
> change locale (en_GB.utf8)
> kill 5492...done
> root@store1:~# ceph -s
>    health HEALTH_OK
>    monmap e1: 3 mons at
> {a=192.168.195.31:6789/0,b=192.168.195.33:6789/0,c=192.168.195.35:6789/0},
> election epoch 82, quorum 0,1,2 a,b,c
>    osdmap e204: 24 osds: 24 up, 24 in
>     pgmap v106709: 5056 pgs: 5056 active+clean; 526 GB data, 1068 GB
> used, 173 TB / 174 TB avail
>    mdsmap e1: 0/0/1 up
> 
> root@store1:~# date -R
> Thu, 25 Apr 2013 13:09:54 +0200
> 
> 
> 
> root@store1:~# ceph -s && date -R
>    health HEALTH_WARN 423 pgs degraded; 423 pgs stuck unclean; recovery
> 10999/269486 degraded (4.081%); 1/24 in osds are down
>    monmap e1: 3 mons at
> {a=192.168.195.31:6789/0,b=192.168.195.33:6789/0,c=192.168.195.35:6789/0},
> election epoch 82, quorum 0,1,2 a,b,c
>    osdmap e206: 24 osds: 23 up, 24 in
>     pgmap v106715: 5056 pgs: 4633 active+clean, 423 active+degraded; 526
> GB data, 1068 GB used, 173 TB / 174 TB avail; 10999/269486 degraded (4.081%)
>    mdsmap e1: 0/0/1 up
> 
> Thu, 25 Apr 2013 13:10:14 +0200
> 
> 
> root@store1:~# ceph -s && date -R
>    health HEALTH_WARN 423 pgs degraded; 423 pgs stuck unclean; recovery
> 10999/269486 degraded (4.081%); 1/24 in osds are down
>    monmap e1: 3 mons at
> {a=192.168.195.31:6789/0,b=192.168.195.33:6789/0,c=192.168.195.35:6789/0},
> election epoch 82, quorum 0,1,2 a,b,c
>    osdmap e206: 24 osds: 23 up, 24 in
>     pgmap v106719: 5056 pgs: 4633 active+clean, 423 active+degraded; 526
> GB data, 1068 GB used, 173 TB / 174 TB avail; 10999/269486 degraded (4.081%)
>    mdsmap e1: 0/0/1 up
> 
> Thu, 25 Apr 2013 13:23:01 +0200
> 
> On 25.04.2013 01:46, Sage Weil wrote:
> > Hi everyone-
> > 
> > We are down to a handful of urgent bugs (3!) and a cuttlefish release date 
> > that is less than a week away.  Thank you to everyone who has been 
> > involved in coding, testing, and stabilizing this release.  We are close!
> > 
> > If you would like to test the current release candidate, your efforts 
> > would be much appreciated!  For deb systems, you can do
> > 
> >  wget -q -O- 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc' | sudo apt-key add -
> >  echo deb http://gitbuilder.ceph.com/ceph-deb-$(lsb_release -sc)-x86_64-basic/ref/next $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
> > 
> > For rpm users you can find packages at
> > 
> >  http://gitbuilder.ceph.com/ceph-rpm-centos6-x86_64-basic/ref/next/
> >  http://gitbuilder.ceph.com/ceph-rpm-fc17-x86_64-basic/ref/next/
> >  http://gitbuilder.ceph.com/ceph-rpm-fc18-x86_64-basic/ref/next/
> > 
> > A draft of the release notes is up at
> > 
> >  http://ceph.com/docs/master/release-notes/#v0-61
> > 
> > Let me know if I've missed anything!
> > 
> > sage
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: cuttlefish countdown -- OSD doesn't get marked out

[Martin Mailand]

Hi Sage,

On 25.04.2013 18:17, Sage Weil wrote:
> What is the output from 'ceph osd tree' and the contents of your 
> [mon*] sections of ceph.conf?
> 
> Thanks!
> sage


root@store1:~# ceph osd tree

# id	weight	type name	up/down	reweight
-1	24	root default
-3	24		rack unknownrack
-2	4			host store1
0	1				osd.0	up	1	
1	1				osd.1	down	1	
2	1				osd.2	up	1	
3	1				osd.3	up	1	
-4	4			host store3
10	1				osd.10	up	1	
11	1				osd.11	up	1	
8	1				osd.8	up	1	
9	1				osd.9	up	1	
-5	4			host store4
12	1				osd.12	up	1	
13	1				osd.13	up	1	
14	1				osd.14	up	1	
15	1				osd.15	up	1	
-6	4			host store5
16	1				osd.16	up	1	
17	1				osd.17	up	1	
18	1				osd.18	up	1	
19	1				osd.19	up	1	
-7	4			host store6
20	1				osd.20	up	1	
21	1				osd.21	up	1	
22	1				osd.22	up	1	
23	1				osd.23	up	1	
-8	4			host store2
4	1				osd.4	up	1	
5	1				osd.5	up	1	
6	1				osd.6	up	1	
7	1				osd.7	up	1	



[global]
        auth cluster requierd = none
        auth service required = none
        auth client required = none
#       log file = ""
        log_max_recent=100
        log_max_new=100

[mon]
        mon data = /data/mon.$id
[mon.a]
        mon host = store1
        mon addr = 192.168.195.31:6789
[mon.b]
        mon host = store3
        mon addr = 192.168.195.33:6789
[mon.c]
        mon host = store5
        mon addr = 192.168.195.35:6789

Re: cuttlefish countdown -- OSD doesn't get marked out

[David Zafman]

I filed tracker bug 4822 and have wip-4822 with a fix.  My manual testing shows that it works.  I'm building a teuthology test.

Given your osd tree has a single rack it should always mark OSDs down after 5 minutes by default.

David Zafman
Senior Developer
http://www.inktank.com




On Apr 25, 2013, at 9:38 AM, Martin Mailand <martin-64xna6+0MClWk0Htik3J/w@public.gmane.org> wrote:

> Hi Sage,
> 
> On 25.04.2013 18:17, Sage Weil wrote:
>> What is the output from 'ceph osd tree' and the contents of your 
>> [mon*] sections of ceph.conf?
>> 
>> Thanks!
>> sage
> 
> 
> root@store1:~# ceph osd tree
> 
> # id	weight	type name	up/down	reweight
> -1	24	root default
> -3	24		rack unknownrack
> -2	4			host store1
> 0	1				osd.0	up	1	
> 1	1				osd.1	down	1	
> 2	1				osd.2	up	1	
> 3	1				osd.3	up	1	
> -4	4			host store3
> 10	1				osd.10	up	1	
> 11	1				osd.11	up	1	
> 8	1				osd.8	up	1	
> 9	1				osd.9	up	1	
> -5	4			host store4
> 12	1				osd.12	up	1	
> 13	1				osd.13	up	1	
> 14	1				osd.14	up	1	
> 15	1				osd.15	up	1	
> -6	4			host store5
> 16	1				osd.16	up	1	
> 17	1				osd.17	up	1	
> 18	1				osd.18	up	1	
> 19	1				osd.19	up	1	
> -7	4			host store6
> 20	1				osd.20	up	1	
> 21	1				osd.21	up	1	
> 22	1				osd.22	up	1	
> 23	1				osd.23	up	1	
> -8	4			host store2
> 4	1				osd.4	up	1	
> 5	1				osd.5	up	1	
> 6	1				osd.6	up	1	
> 7	1				osd.7	up	1	
> 
> 
> 
> [global]
>        auth cluster requierd = none
>        auth service required = none
>        auth client required = none
> #       log file = ""
>        log_max_recent=100
>        log_max_new=100
> 
> [mon]
>        mon data = /data/mon.$id
> [mon.a]
>        mon host = store1
>        mon addr = 192.168.195.31:6789
> [mon.b]
>        mon host = store3
>        mon addr = 192.168.195.33:6789
> [mon.c]
>        mon host = store5
>        mon addr = 192.168.195.35:6789
> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cuttlefish countdown -- OSD doesn't get marked out

[Martin Mailand]

Hi David,

did you test it with more than one rack as well? In my first problem I
used two racks, with a custom crushmap, so that the replicas are in the
two racks (replicationlevel = 2). Than I took one osd down, and expected
that the remaining osds in this rack would get the now missing replicas
from the osd of the other rack.
But nothing happened, the cluster stayed degraded.

-martin


On 26.04.2013 02:22, David Zafman wrote:
> 
> I filed tracker bug 4822 and have wip-4822 with a fix.  My manual testing shows that it works.  I'm building a teuthology test.
> 
> Given your osd tree has a single rack it should always mark OSDs down after 5 minutes by default.
> 
> David Zafman
> Senior Developer
> http://www.inktank.com
> 
> 
> 
> 
> On Apr 25, 2013, at 9:38 AM, Martin Mailand <martin@tuxadero.com> wrote:
> 
>> Hi Sage,
>>
>> On 25.04.2013 18:17, Sage Weil wrote:
>>> What is the output from 'ceph osd tree' and the contents of your 
>>> [mon*] sections of ceph.conf?
>>>
>>> Thanks!
>>> sage
>>
>>
>> root@store1:~# ceph osd tree
>>
>> # id	weight	type name	up/down	reweight
>> -1	24	root default
>> -3	24		rack unknownrack
>> -2	4			host store1
>> 0	1				osd.0	up	1	
>> 1	1				osd.1	down	1	
>> 2	1				osd.2	up	1	
>> 3	1				osd.3	up	1	
>> -4	4			host store3
>> 10	1				osd.10	up	1	
>> 11	1				osd.11	up	1	
>> 8	1				osd.8	up	1	
>> 9	1				osd.9	up	1	
>> -5	4			host store4
>> 12	1				osd.12	up	1	
>> 13	1				osd.13	up	1	
>> 14	1				osd.14	up	1	
>> 15	1				osd.15	up	1	
>> -6	4			host store5
>> 16	1				osd.16	up	1	
>> 17	1				osd.17	up	1	
>> 18	1				osd.18	up	1	
>> 19	1				osd.19	up	1	
>> -7	4			host store6
>> 20	1				osd.20	up	1	
>> 21	1				osd.21	up	1	
>> 22	1				osd.22	up	1	
>> 23	1				osd.23	up	1	
>> -8	4			host store2
>> 4	1				osd.4	up	1	
>> 5	1				osd.5	up	1	
>> 6	1				osd.6	up	1	
>> 7	1				osd.7	up	1	
>>
>>
>>
>> [global]
>>        auth cluster requierd = none
>>        auth service required = none
>>        auth client required = none
>> #       log file = ""
>>        log_max_recent=100
>>        log_max_new=100
>>
>> [mon]
>>        mon data = /data/mon.$id
>> [mon.a]
>>        mon host = store1
>>        mon addr = 192.168.195.31:6789
>> [mon.b]
>>        mon host = store3
>>        mon addr = 192.168.195.33:6789
>> [mon.c]
>>        mon host = store5
>>        mon addr = 192.168.195.35:6789
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

Re: [ceph-users] cuttlefish countdown -- OSD doesn't get marked out

[Mike Dawson]

David / Martin,

I can confirm this issue. At present I am running monitors only with 
100% of my OSD processes shutdown down. For the past couple hours, Ceph 
has reported:

osdmap e1323: 66 osds: 19 up, 66 in

I can mark them down manually using

ceph osd down 0

as expected, but they never get marked down automatically. Like Martin, 
I also have a custom crushmap, but this cluster is operating with a 
single rack. I'll be happy to provide any documentation / configs / logs 
you would like.

I am currently running ceph version 0.60-666-ga5cade1 
(a5cade1fe7338602fb2bbfa867433d825f337c87) from gitbuilder.

- Mike

On 4/26/2013 4:50 AM, Martin Mailand wrote:
> Hi David,
>
> did you test it with more than one rack as well? In my first problem I
> used two racks, with a custom crushmap, so that the replicas are in the
> two racks (replicationlevel = 2). Than I took one osd down, and expected
> that the remaining osds in this rack would get the now missing replicas
> from the osd of the other rack.
> But nothing happened, the cluster stayed degraded.
>
> -martin
>
>
> On 26.04.2013 02:22, David Zafman wrote:
>>
>> I filed tracker bug 4822 and have wip-4822 with a fix.  My manual testing shows that it works.  I'm building a teuthology test.
>>
>> Given your osd tree has a single rack it should always mark OSDs down after 5 minutes by default.
>>
>> David Zafman
>> Senior Developer
>> http://www.inktank.com
>>
>>
>>
>>
>> On Apr 25, 2013, at 9:38 AM, Martin Mailand <martin@tuxadero.com> wrote:
>>
>>> Hi Sage,
>>>
>>> On 25.04.2013 18:17, Sage Weil wrote:
>>>> What is the output from 'ceph osd tree' and the contents of your
>>>> [mon*] sections of ceph.conf?
>>>>
>>>> Thanks!
>>>> sage
>>>
>>>
>>> root@store1:~# ceph osd tree
>>>
>>> # id	weight	type name	up/down	reweight
>>> -1	24	root default
>>> -3	24		rack unknownrack
>>> -2	4			host store1
>>> 0	1				osd.0	up	1	
>>> 1	1				osd.1	down	1	
>>> 2	1				osd.2	up	1	
>>> 3	1				osd.3	up	1	
>>> -4	4			host store3
>>> 10	1				osd.10	up	1	
>>> 11	1				osd.11	up	1	
>>> 8	1				osd.8	up	1	
>>> 9	1				osd.9	up	1	
>>> -5	4			host store4
>>> 12	1				osd.12	up	1	
>>> 13	1				osd.13	up	1	
>>> 14	1				osd.14	up	1	
>>> 15	1				osd.15	up	1	
>>> -6	4			host store5
>>> 16	1				osd.16	up	1	
>>> 17	1				osd.17	up	1	
>>> 18	1				osd.18	up	1	
>>> 19	1				osd.19	up	1	
>>> -7	4			host store6
>>> 20	1				osd.20	up	1	
>>> 21	1				osd.21	up	1	
>>> 22	1				osd.22	up	1	
>>> 23	1				osd.23	up	1	
>>> -8	4			host store2
>>> 4	1				osd.4	up	1	
>>> 5	1				osd.5	up	1	
>>> 6	1				osd.6	up	1	
>>> 7	1				osd.7	up	1	
>>>
>>>
>>>
>>> [global]
>>>         auth cluster requierd = none
>>>         auth service required = none
>>>         auth client required = none
>>> #       log file = ""
>>>         log_max_recent=100
>>>         log_max_new=100
>>>
>>> [mon]
>>>         mon data = /data/mon.$id
>>> [mon.a]
>>>         mon host = store1
>>>         mon addr = 192.168.195.31:6789
>>> [mon.b]
>>>         mon host = store3
>>>         mon addr = 192.168.195.33:6789
>>> [mon.c]
>>>         mon host = store5
>>>         mon addr = 192.168.195.35:6789
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: [ceph-users] cuttlefish countdown -- OSD doesn't get marked out

[David Zafman]

Mike / Martin,

The OSD down behavior Mike is seeing is different.  You should be seeing messages like this in your leader's monitor log:

can_mark_down current up_ratio 0.166667 < min 0.3, will not mark osd.2 down

To dampen certain kinds of cascading failures, we are deliberately restricting automatically marking < 30% of OSDs down.

As far as Martin is concerned his osd tree shows a single rack, but said that his crush rules are supposed to put a replica on each of 2 racks.  I don't remember seeing his crush rules in any of the e-mails, but even so he only has "unknownrack" with id -3 defined.

David Zafman
Senior Developer
http://www.inktank.com




On Apr 26, 2013, at 6:44 AM, Mike Dawson <mike.dawson@scholarstack.com> wrote:

> David / Martin,
> 
> I can confirm this issue. At present I am running monitors only with 100% of my OSD processes shutdown down. For the past couple hours, Ceph has reported:
> 
> osdmap e1323: 66 osds: 19 up, 66 in
> 
> I can mark them down manually using
> 
> ceph osd down 0
> 
> as expected, but they never get marked down automatically. Like Martin, I also have a custom crushmap, but this cluster is operating with a single rack. I'll be happy to provide any documentation / configs / logs you would like.
> 
> I am currently running ceph version 0.60-666-ga5cade1 (a5cade1fe7338602fb2bbfa867433d825f337c87) from gitbuilder.
> 
> - Mike
> 
> On 4/26/2013 4:50 AM, Martin Mailand wrote:
>> Hi David,
>> 
>> did you test it with more than one rack as well? In my first problem I
>> used two racks, with a custom crushmap, so that the replicas are in the
>> two racks (replicationlevel = 2). Than I took one osd down, and expected
>> that the remaining osds in this rack would get the now missing replicas
>> from the osd of the other rack.
>> But nothing happened, the cluster stayed degraded.
>> 
>> -martin
>> 
>> 
>> On 26.04.2013 02:22, David Zafman wrote:
>>> 
>>> I filed tracker bug 4822 and have wip-4822 with a fix.  My manual testing shows that it works.  I'm building a teuthology test.
>>> 
>>> Given your osd tree has a single rack it should always mark OSDs down after 5 minutes by default.
>>> 
>>> David Zafman
>>> Senior Developer
>>> http://www.inktank.com
>>> 
>>> 
>>> 
>>> 
>>> On Apr 25, 2013, at 9:38 AM, Martin Mailand <martin@tuxadero.com> wrote:
>>> 
>>>> Hi Sage,
>>>> 
>>>> On 25.04.2013 18:17, Sage Weil wrote:
>>>>> What is the output from 'ceph osd tree' and the contents of your
>>>>> [mon*] sections of ceph.conf?
>>>>> 
>>>>> Thanks!
>>>>> sage
>>>> 
>>>> 
>>>> root@store1:~# ceph osd tree
>>>> 
>>>> # id	weight	type name	up/down	reweight
>>>> -1	24	root default
>>>> -3	24		rack unknownrack
>>>> -2	4			host store1
>>>> 0	1				osd.0	up	1	
>>>> 1	1				osd.1	down	1	
>>>> 2	1				osd.2	up	1	
>>>> 3	1				osd.3	up	1	
>>>> -4	4			host store3
>>>> 10	1				osd.10	up	1	
>>>> 11	1				osd.11	up	1	
>>>> 8	1				osd.8	up	1	
>>>> 9	1				osd.9	up	1	
>>>> -5	4			host store4
>>>> 12	1				osd.12	up	1	
>>>> 13	1				osd.13	up	1	
>>>> 14	1				osd.14	up	1	
>>>> 15	1				osd.15	up	1	
>>>> -6	4			host store5
>>>> 16	1				osd.16	up	1	
>>>> 17	1				osd.17	up	1	
>>>> 18	1				osd.18	up	1	
>>>> 19	1				osd.19	up	1	
>>>> -7	4			host store6
>>>> 20	1				osd.20	up	1	
>>>> 21	1				osd.21	up	1	
>>>> 22	1				osd.22	up	1	
>>>> 23	1				osd.23	up	1	
>>>> -8	4			host store2
>>>> 4	1				osd.4	up	1	
>>>> 5	1				osd.5	up	1	
>>>> 6	1				osd.6	up	1	
>>>> 7	1				osd.7	up	1	
>>>> 
>>>> 
>>>> 
>>>> [global]
>>>>        auth cluster requierd = none
>>>>        auth service required = none
>>>>        auth client required = none
>>>> #       log file = ""
>>>>        log_max_recent=100
>>>>        log_max_new=100
>>>> 
>>>> [mon]
>>>>        mon data = /data/mon.$id
>>>> [mon.a]
>>>>        mon host = store1
>>>>        mon addr = 192.168.195.31:6789
>>>> [mon.b]
>>>>        mon host = store3
>>>>        mon addr = 192.168.195.33:6789
>>>> [mon.c]
>>>>        mon host = store5
>>>>        mon addr = 192.168.195.35:6789
>>>> _______________________________________________
>>>> ceph-users mailing list
>>>> ceph-users@lists.ceph.com
>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> 
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> 

Re: [ceph-users] cuttlefish countdown -- OSD doesn't get marked out

[David Zafman]

The behavior you are seeing is exactly what would be expected if OSDs are not being marked out.  The testing of my fix showed that if a portion of a rack's OSDs go down they will be marked out after the configured amount of time (5 min by default).  Once down OSDs are out the remaining OSDs take responsibility for holding the data assigned to that rack.

Though I didn't look at the data movement, I'm confident that it will work.  You can simply mark your OSDs out manually to verify that missing replicas are replaced.

David Zafman
Senior Developer
http://www.inktank.com




On Apr 26, 2013, at 1:50 AM, Martin Mailand <martin@tuxadero.com> wrote:

> Hi David,
> 
> did you test it with more than one rack as well? In my first problem I
> used two racks, with a custom crushmap, so that the replicas are in the
> two racks (replicationlevel = 2). Than I took one osd down, and expected
> that the remaining osds in this rack would get the now missing replicas
> from the osd of the other rack.
> But nothing happened, the cluster stayed degraded.
> 
> -martin
> 
> 
> On 26.04.2013 02:22, David Zafman wrote:
>> 
>> I filed tracker bug 4822 and have wip-4822 with a fix.  My manual testing shows that it works.  I'm building a teuthology test.
>> 
>> Given your osd tree has a single rack it should always mark OSDs down after 5 minutes by default.
>> 
>> David Zafman
>> Senior Developer
>> http://www.inktank.com
>> 
>> 
>> 
>> 
>> On Apr 25, 2013, at 9:38 AM, Martin Mailand <martin@tuxadero.com> wrote:
>> 
>>> Hi Sage,
>>> 
>>> On 25.04.2013 18:17, Sage Weil wrote:
>>>> What is the output from 'ceph osd tree' and the contents of your 
>>>> [mon*] sections of ceph.conf?
>>>> 
>>>> Thanks!
>>>> sage
>>> 
>>> 
>>> root@store1:~# ceph osd tree
>>> 
>>> # id	weight	type name	up/down	reweight
>>> -1	24	root default
>>> -3	24		rack unknownrack
>>> -2	4			host store1
>>> 0	1				osd.0	up	1	
>>> 1	1				osd.1	down	1	
>>> 2	1				osd.2	up	1	
>>> 3	1				osd.3	up	1	
>>> -4	4			host store3
>>> 10	1				osd.10	up	1	
>>> 11	1				osd.11	up	1	
>>> 8	1				osd.8	up	1	
>>> 9	1				osd.9	up	1	
>>> -5	4			host store4
>>> 12	1				osd.12	up	1	
>>> 13	1				osd.13	up	1	
>>> 14	1				osd.14	up	1	
>>> 15	1				osd.15	up	1	
>>> -6	4			host store5
>>> 16	1				osd.16	up	1	
>>> 17	1				osd.17	up	1	
>>> 18	1				osd.18	up	1	
>>> 19	1				osd.19	up	1	
>>> -7	4			host store6
>>> 20	1				osd.20	up	1	
>>> 21	1				osd.21	up	1	
>>> 22	1				osd.22	up	1	
>>> 23	1				osd.23	up	1	
>>> -8	4			host store2
>>> 4	1				osd.4	up	1	
>>> 5	1				osd.5	up	1	
>>> 6	1				osd.6	up	1	
>>> 7	1				osd.7	up	1	
>>> 
>>> 
>>> 
>>> [global]
>>>       auth cluster requierd = none
>>>       auth service required = none
>>>       auth client required = none
>>> #       log file = ""
>>>       log_max_recent=100
>>>       log_max_new=100
>>> 
>>> [mon]
>>>       mon data = /data/mon.$id
>>> [mon.a]
>>>       mon host = store1
>>>       mon addr = 192.168.195.31:6789
>>> [mon.b]
>>>       mon host = store3
>>>       mon addr = 192.168.195.33:6789
>>> [mon.c]
>>>       mon host = store5
>>>       mon addr = 192.168.195.35:6789
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> 

Re: cuttlefish countdown -- OSD doesn't get marked out

[Mike Dawson]

Sage,

I confirm this issue. The requested info is listed below.

*Note that due to the pre-Cuttlefish monitor sync issues, this 
deployment has been running three monitors (mon.b and mon.c working 
properly in quorum. mon.a stuck forever synchronizing).

For the past two hours, no OSD processes have been running on any host, 
yet some OSDs are still marked as up.

http://www.gammacode.com/upload/ceph-osd-tree


The mon* sections of ceph.conf are:

[mon]
     debug mon = 20
     debug paxos = 20
     debug ms = 1

[mon.a]
     host = node2
     mon addr = 10.1.0.3:6789

[mon.b]
     host = node26
     mon addr = 10.1.0.67:6789

[mon.c]
     host = node49
     mon addr = 10.1.0.130:6789

root@controller1:~# ceph -s
    health HEALTH_WARN 43 pgs degraded; 13308 pgs peering; 27932 pgs 
stale; 13308 pgs stuck inactive; 27932 pgs stuck stale; 13582 pgs stuck 
unclean; recovery 7264/7986546 degraded (0.091%); 47/66 in osds are 
down; 1 mons down, quorum 1,2 b,c
    monmap e1: 3 mons at 
{a=10.1.0.3:6789/0,b=10.1.0.67:6789/0,c=10.1.0.130:6789/0}, election 
epoch 1428, quorum 1,2 b,c
    osdmap e1323: 66 osds: 19 up, 66 in
     pgmap v427324: 28864 pgs: 257 active+clean, 231 stale+active, 15025 
stale+active+clean, 675 peering, 12633 stale+peering, 43 
stale+active+degraded; 448 GB data, 1402 GB used, 178 TB / 180 TB avail; 
7264/7986546 degraded (0.091%)
    mdsmap e1: 0/0/1 up

For reference, this is ceph version 0.60-666-ga5cade1 
(a5cade1fe7338602fb2bbfa867433d825f337c87) from gitbuilder.

Thanks,
Mike

On 4/25/2013 12:17 PM, Sage Weil wrote:
> On Thu, 25 Apr 2013, Martin Mailand wrote:
>> Hi,
>>
>> if I shutdown an OSD, the OSD gets marked down after 20 seconds, after
>> 300 seconds the osd should get marked out, an the cluster should resync.
>> But that doesn't happened, the OSD stays in the status down/in forever,
>> therefore the cluster stays forever degraded.
>> I can reproduce it with a new installed cluster.
>>
>> If I manually set the osd out (ceph osd out 1), the cluster resync
>> starts immediately.
>>
>> I think thats a release critical bug, because the cluster health is not
>> automatically recovered.
>
> What is the output from 'ceph osd tree' and the contents of your
> [mon*] sections of ceph.conf?
>
> Thanks!
> sage
>
>
>>
>> And I reported this behavior a while ago
>> http://article.gmane.org/gmane.comp.file-systems.ceph.user/603/
>>
>> -martin
>>
>>
>> Log:
>>
>>
>> root@store1:~# ceph -s
>>     health HEALTH_OK
>>     monmap e1: 3 mons at
>> {a=192.168.195.31:6789/0,b=192.168.195.33:6789/0,c=192.168.195.35:6789/0},
>> election epoch 82, quorum 0,1,2 a,b,c
>>     osdmap e204: 24 osds: 24 up, 24 in
>>      pgmap v106709: 5056 pgs: 5056 active+clean; 526 GB data, 1068 GB
>> used, 173 TB / 174 TB avail
>>     mdsmap e1: 0/0/1 up
>>
>> root@store1:~# ceph --version
>> ceph version 0.60 (f26f7a39021dbf440c28d6375222e21c94fe8e5c)
>> root@store1:~# /etc/init.d/ceph stop osd.1
>> === osd.1 ===
>> Stopping Ceph osd.1 on store1...bash: warning: setlocale: LC_ALL: cannot
>> change locale (en_GB.utf8)
>> kill 5492...done
>> root@store1:~# ceph -s
>>     health HEALTH_OK
>>     monmap e1: 3 mons at
>> {a=192.168.195.31:6789/0,b=192.168.195.33:6789/0,c=192.168.195.35:6789/0},
>> election epoch 82, quorum 0,1,2 a,b,c
>>     osdmap e204: 24 osds: 24 up, 24 in
>>      pgmap v106709: 5056 pgs: 5056 active+clean; 526 GB data, 1068 GB
>> used, 173 TB / 174 TB avail
>>     mdsmap e1: 0/0/1 up
>>
>> root@store1:~# date -R
>> Thu, 25 Apr 2013 13:09:54 +0200
>>
>>
>>
>> root@store1:~# ceph -s && date -R
>>     health HEALTH_WARN 423 pgs degraded; 423 pgs stuck unclean; recovery
>> 10999/269486 degraded (4.081%); 1/24 in osds are down
>>     monmap e1: 3 mons at
>> {a=192.168.195.31:6789/0,b=192.168.195.33:6789/0,c=192.168.195.35:6789/0},
>> election epoch 82, quorum 0,1,2 a,b,c
>>     osdmap e206: 24 osds: 23 up, 24 in
>>      pgmap v106715: 5056 pgs: 4633 active+clean, 423 active+degraded; 526
>> GB data, 1068 GB used, 173 TB / 174 TB avail; 10999/269486 degraded (4.081%)
>>     mdsmap e1: 0/0/1 up
>>
>> Thu, 25 Apr 2013 13:10:14 +0200
>>
>>
>> root@store1:~# ceph -s && date -R
>>     health HEALTH_WARN 423 pgs degraded; 423 pgs stuck unclean; recovery
>> 10999/269486 degraded (4.081%); 1/24 in osds are down
>>     monmap e1: 3 mons at
>> {a=192.168.195.31:6789/0,b=192.168.195.33:6789/0,c=192.168.195.35:6789/0},
>> election epoch 82, quorum 0,1,2 a,b,c
>>     osdmap e206: 24 osds: 23 up, 24 in
>>      pgmap v106719: 5056 pgs: 4633 active+clean, 423 active+degraded; 526
>> GB data, 1068 GB used, 173 TB / 174 TB avail; 10999/269486 degraded (4.081%)
>>     mdsmap e1: 0/0/1 up
>>
>> Thu, 25 Apr 2013 13:23:01 +0200
>>
>> On 25.04.2013 01:46, Sage Weil wrote:
>>> Hi everyone-
>>>
>>> We are down to a handful of urgent bugs (3!) and a cuttlefish release date
>>> that is less than a week away.  Thank you to everyone who has been
>>> involved in coding, testing, and stabilizing this release.  We are close!
>>>
>>> If you would like to test the current release candidate, your efforts
>>> would be much appreciated!  For deb systems, you can do
>>>
>>>   wget -q -O- 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc' | sudo apt-key add -
>>>   echo deb http://gitbuilder.ceph.com/ceph-deb-$(lsb_release -sc)-x86_64-basic/ref/next $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
>>>
>>> For rpm users you can find packages at
>>>
>>>   http://gitbuilder.ceph.com/ceph-rpm-centos6-x86_64-basic/ref/next/
>>>   http://gitbuilder.ceph.com/ceph-rpm-fc17-x86_64-basic/ref/next/
>>>   http://gitbuilder.ceph.com/ceph-rpm-fc18-x86_64-basic/ref/next/
>>>
>>> A draft of the release notes is up at
>>>
>>>   http://ceph.com/docs/master/release-notes/#v0-61
>>>
>>> Let me know if I've missed anything!
>>>
>>> sage
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>