* cuttlefish countdown
@ 2013-04-24 23:46 Sage Weil
2013-04-25 8:17 ` Wido den Hollander
2013-04-25 12:07 ` cuttlefish countdown -- OSD doesn't get marked out Martin Mailand
0 siblings, 2 replies; 16+ messages in thread
From: Sage Weil @ 2013-04-24 23:46 UTC (permalink / raw)
To: ceph-devel-u79uwXL29TY76Z2rM5mHXA, ceph-users-Qp0mS5GaXlQ
Hi everyone-
We are down to a handful of urgent bugs (3!) and a cuttlefish release date
that is less than a week away. Thank you to everyone who has been
involved in coding, testing, and stabilizing this release. We are close!
If you would like to test the current release candidate, your efforts
would be much appreciated! For deb systems, you can do
wget -q -O- 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc' | sudo apt-key add -
echo deb http://gitbuilder.ceph.com/ceph-deb-$(lsb_release -sc)-x86_64-basic/ref/next $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
For rpm users you can find packages at
http://gitbuilder.ceph.com/ceph-rpm-centos6-x86_64-basic/ref/next/
http://gitbuilder.ceph.com/ceph-rpm-fc17-x86_64-basic/ref/next/
http://gitbuilder.ceph.com/ceph-rpm-fc18-x86_64-basic/ref/next/
A draft of the release notes is up at
http://ceph.com/docs/master/release-notes/#v0-61
Let me know if I've missed anything!
sage
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: cuttlefish countdown
2013-04-24 23:46 cuttlefish countdown Sage Weil
@ 2013-04-25 8:17 ` Wido den Hollander
2013-04-25 10:03 ` Kevin Decherf
2013-04-25 16:13 ` Sage Weil
2013-04-25 12:07 ` cuttlefish countdown -- OSD doesn't get marked out Martin Mailand
1 sibling, 2 replies; 16+ messages in thread
From: Wido den Hollander @ 2013-04-25 8:17 UTC (permalink / raw)
To: Sage Weil; +Cc: ceph-devel
Hi,
On 04/25/2013 01:46 AM, Sage Weil wrote:
> Hi everyone-
>
> We are down to a handful of urgent bugs (3!) and a cuttlefish release date
> that is less than a week away. Thank you to everyone who has been
> involved in coding, testing, and stabilizing this release. We are close!
>
I'm trying to find the 3 urgent bugs for Cuttlefish, but I can't seem to
fetch them out of Redmine:
http://tracker.ceph.com/projects/1/issues?sort=position
That doesn't show me any open urgent bugs? Trying to figure out what's
still open.
Wido
> If you would like to test the current release candidate, your efforts
> would be much appreciated! For deb systems, you can do
>
> wget -q -O- 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc' | sudo apt-key add -
> echo deb http://gitbuilder.ceph.com/ceph-deb-$(lsb_release -sc)-x86_64-basic/ref/next $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
>
> For rpm users you can find packages at
>
> http://gitbuilder.ceph.com/ceph-rpm-centos6-x86_64-basic/ref/next/
> http://gitbuilder.ceph.com/ceph-rpm-fc17-x86_64-basic/ref/next/
> http://gitbuilder.ceph.com/ceph-rpm-fc18-x86_64-basic/ref/next/
>
> A draft of the release notes is up at
>
> http://ceph.com/docs/master/release-notes/#v0-61
>
> Let me know if I've missed anything!
>
> sage
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
Wido den Hollander
42on B.V.
Phone: +31 (0)20 700 9902
Skype: contact42on
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: cuttlefish countdown
2013-04-25 8:17 ` Wido den Hollander
@ 2013-04-25 10:03 ` Kevin Decherf
2013-04-25 16:05 ` Ian Colle
2013-04-25 16:13 ` Sage Weil
1 sibling, 1 reply; 16+ messages in thread
From: Kevin Decherf @ 2013-04-25 10:03 UTC (permalink / raw)
To: Wido den Hollander; +Cc: Sage Weil, ceph-devel
Hi,
On Thu, Apr 25, 2013 at 10:17:47AM +0200, Wido den Hollander wrote:
> Hi,
>
> On 04/25/2013 01:46 AM, Sage Weil wrote:
> >Hi everyone-
> >
> >We are down to a handful of urgent bugs (3!) and a cuttlefish release date
> >that is less than a week away. Thank you to everyone who has been
> >involved in coding, testing, and stabilizing this release. We are close!
> >
>
> I'm trying to find the 3 urgent bugs for Cuttlefish, but I can't
> seem to fetch them out of Redmine:
> http://tracker.ceph.com/projects/1/issues?sort=position
>
> That doesn't show me any open urgent bugs? Trying to figure out
> what's still open.
http://tracker.ceph.com/projects/ceph/issues?utf8=%E2%9C%93&set_filter=1&f[]=status_id&op[status_id]=o&f[]=fixed_version_id&op[fixed_version_id]=%3D&v[fixed_version_id][]=138&f[]=&c[]=project&c[]=tracker&c[]=status&c[]=priority&c[]=subject&c[]=assigned_to&c[]=updated_on&c[]=category&c[]=fixed_version&c[]=story_points&c[]=cf_3&c[]=cf_4&group_by=
That's what you are looking for?
--
Kevin Decherf - @Kdecherf
GPG C610 FE73 E706 F968 612B E4B2 108A BD75 A81E 6E2F
http://kdecherf.com
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: cuttlefish countdown -- OSD doesn't get marked out
2013-04-24 23:46 cuttlefish countdown Sage Weil
2013-04-25 8:17 ` Wido den Hollander
@ 2013-04-25 12:07 ` Martin Mailand
2013-04-25 12:56 ` Wido den Hollander
2013-04-25 16:17 ` Sage Weil
1 sibling, 2 replies; 16+ messages in thread
From: Martin Mailand @ 2013-04-25 12:07 UTC (permalink / raw)
To: Sage Weil; +Cc: ceph-devel, ceph-users
Hi,
if I shutdown an OSD, the OSD gets marked down after 20 seconds, after
300 seconds the osd should get marked out, an the cluster should resync.
But that doesn't happened, the OSD stays in the status down/in forever,
therefore the cluster stays forever degraded.
I can reproduce it with a new installed cluster.
If I manually set the osd out (ceph osd out 1), the cluster resync
starts immediately.
I think thats a release critical bug, because the cluster health is not
automatically recovered.
And I reported this behavior a while ago
http://article.gmane.org/gmane.comp.file-systems.ceph.user/603/
-martin
Log:
root@store1:~# ceph -s
health HEALTH_OK
monmap e1: 3 mons at
{a=192.168.195.31:6789/0,b=192.168.195.33:6789/0,c=192.168.195.35:6789/0},
election epoch 82, quorum 0,1,2 a,b,c
osdmap e204: 24 osds: 24 up, 24 in
pgmap v106709: 5056 pgs: 5056 active+clean; 526 GB data, 1068 GB
used, 173 TB / 174 TB avail
mdsmap e1: 0/0/1 up
root@store1:~# ceph --version
ceph version 0.60 (f26f7a39021dbf440c28d6375222e21c94fe8e5c)
root@store1:~# /etc/init.d/ceph stop osd.1
=== osd.1 ===
Stopping Ceph osd.1 on store1...bash: warning: setlocale: LC_ALL: cannot
change locale (en_GB.utf8)
kill 5492...done
root@store1:~# ceph -s
health HEALTH_OK
monmap e1: 3 mons at
{a=192.168.195.31:6789/0,b=192.168.195.33:6789/0,c=192.168.195.35:6789/0},
election epoch 82, quorum 0,1,2 a,b,c
osdmap e204: 24 osds: 24 up, 24 in
pgmap v106709: 5056 pgs: 5056 active+clean; 526 GB data, 1068 GB
used, 173 TB / 174 TB avail
mdsmap e1: 0/0/1 up
root@store1:~# date -R
Thu, 25 Apr 2013 13:09:54 +0200
root@store1:~# ceph -s && date -R
health HEALTH_WARN 423 pgs degraded; 423 pgs stuck unclean; recovery
10999/269486 degraded (4.081%); 1/24 in osds are down
monmap e1: 3 mons at
{a=192.168.195.31:6789/0,b=192.168.195.33:6789/0,c=192.168.195.35:6789/0},
election epoch 82, quorum 0,1,2 a,b,c
osdmap e206: 24 osds: 23 up, 24 in
pgmap v106715: 5056 pgs: 4633 active+clean, 423 active+degraded; 526
GB data, 1068 GB used, 173 TB / 174 TB avail; 10999/269486 degraded (4.081%)
mdsmap e1: 0/0/1 up
Thu, 25 Apr 2013 13:10:14 +0200
root@store1:~# ceph -s && date -R
health HEALTH_WARN 423 pgs degraded; 423 pgs stuck unclean; recovery
10999/269486 degraded (4.081%); 1/24 in osds are down
monmap e1: 3 mons at
{a=192.168.195.31:6789/0,b=192.168.195.33:6789/0,c=192.168.195.35:6789/0},
election epoch 82, quorum 0,1,2 a,b,c
osdmap e206: 24 osds: 23 up, 24 in
pgmap v106719: 5056 pgs: 4633 active+clean, 423 active+degraded; 526
GB data, 1068 GB used, 173 TB / 174 TB avail; 10999/269486 degraded (4.081%)
mdsmap e1: 0/0/1 up
Thu, 25 Apr 2013 13:23:01 +0200
On 25.04.2013 01:46, Sage Weil wrote:
> Hi everyone-
>
> We are down to a handful of urgent bugs (3!) and a cuttlefish release date
> that is less than a week away. Thank you to everyone who has been
> involved in coding, testing, and stabilizing this release. We are close!
>
> If you would like to test the current release candidate, your efforts
> would be much appreciated! For deb systems, you can do
>
> wget -q -O- 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc' | sudo apt-key add -
> echo deb http://gitbuilder.ceph.com/ceph-deb-$(lsb_release -sc)-x86_64-basic/ref/next $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
>
> For rpm users you can find packages at
>
> http://gitbuilder.ceph.com/ceph-rpm-centos6-x86_64-basic/ref/next/
> http://gitbuilder.ceph.com/ceph-rpm-fc17-x86_64-basic/ref/next/
> http://gitbuilder.ceph.com/ceph-rpm-fc18-x86_64-basic/ref/next/
>
> A draft of the release notes is up at
>
> http://ceph.com/docs/master/release-notes/#v0-61
>
> Let me know if I've missed anything!
>
> sage
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: cuttlefish countdown -- OSD doesn't get marked out
2013-04-25 12:07 ` cuttlefish countdown -- OSD doesn't get marked out Martin Mailand
@ 2013-04-25 12:56 ` Wido den Hollander
2013-04-25 13:04 ` Martin Mailand
2013-04-25 16:17 ` Sage Weil
1 sibling, 1 reply; 16+ messages in thread
From: Wido den Hollander @ 2013-04-25 12:56 UTC (permalink / raw)
To: Martin Mailand; +Cc: Sage Weil, ceph-devel
On 04/25/2013 02:07 PM, Martin Mailand wrote:
> Hi,
>
> if I shutdown an OSD, the OSD gets marked down after 20 seconds, after
> 300 seconds the osd should get marked out, an the cluster should resync.
> But that doesn't happened, the OSD stays in the status down/in forever,
> therefore the cluster stays forever degraded.
> I can reproduce it with a new installed cluster.
>
> If I manually set the osd out (ceph osd out 1), the cluster resync
> starts immediately.
>
Could you dump your osdmap? The first 10 lines would be interesting.
There is a flag where you say "noosdout", could it be that the flag is set?
Wido
> I think thats a release critical bug, because the cluster health is not
> automatically recovered.
>
> And I reported this behavior a while ago
> http://article.gmane.org/gmane.comp.file-systems.ceph.user/603/
>
> -martin
>
>
> Log:
>
>
> root@store1:~# ceph -s
> health HEALTH_OK
> monmap e1: 3 mons at
> {a=192.168.195.31:6789/0,b=192.168.195.33:6789/0,c=192.168.195.35:6789/0},
> election epoch 82, quorum 0,1,2 a,b,c
> osdmap e204: 24 osds: 24 up, 24 in
> pgmap v106709: 5056 pgs: 5056 active+clean; 526 GB data, 1068 GB
> used, 173 TB / 174 TB avail
> mdsmap e1: 0/0/1 up
>
> root@store1:~# ceph --version
> ceph version 0.60 (f26f7a39021dbf440c28d6375222e21c94fe8e5c)
> root@store1:~# /etc/init.d/ceph stop osd.1
> === osd.1 ===
> Stopping Ceph osd.1 on store1...bash: warning: setlocale: LC_ALL: cannot
> change locale (en_GB.utf8)
> kill 5492...done
> root@store1:~# ceph -s
> health HEALTH_OK
> monmap e1: 3 mons at
> {a=192.168.195.31:6789/0,b=192.168.195.33:6789/0,c=192.168.195.35:6789/0},
> election epoch 82, quorum 0,1,2 a,b,c
> osdmap e204: 24 osds: 24 up, 24 in
> pgmap v106709: 5056 pgs: 5056 active+clean; 526 GB data, 1068 GB
> used, 173 TB / 174 TB avail
> mdsmap e1: 0/0/1 up
>
> root@store1:~# date -R
> Thu, 25 Apr 2013 13:09:54 +0200
>
>
>
> root@store1:~# ceph -s && date -R
> health HEALTH_WARN 423 pgs degraded; 423 pgs stuck unclean; recovery
> 10999/269486 degraded (4.081%); 1/24 in osds are down
> monmap e1: 3 mons at
> {a=192.168.195.31:6789/0,b=192.168.195.33:6789/0,c=192.168.195.35:6789/0},
> election epoch 82, quorum 0,1,2 a,b,c
> osdmap e206: 24 osds: 23 up, 24 in
> pgmap v106715: 5056 pgs: 4633 active+clean, 423 active+degraded; 526
> GB data, 1068 GB used, 173 TB / 174 TB avail; 10999/269486 degraded (4.081%)
> mdsmap e1: 0/0/1 up
>
> Thu, 25 Apr 2013 13:10:14 +0200
>
>
> root@store1:~# ceph -s && date -R
> health HEALTH_WARN 423 pgs degraded; 423 pgs stuck unclean; recovery
> 10999/269486 degraded (4.081%); 1/24 in osds are down
> monmap e1: 3 mons at
> {a=192.168.195.31:6789/0,b=192.168.195.33:6789/0,c=192.168.195.35:6789/0},
> election epoch 82, quorum 0,1,2 a,b,c
> osdmap e206: 24 osds: 23 up, 24 in
> pgmap v106719: 5056 pgs: 4633 active+clean, 423 active+degraded; 526
> GB data, 1068 GB used, 173 TB / 174 TB avail; 10999/269486 degraded (4.081%)
> mdsmap e1: 0/0/1 up
>
> Thu, 25 Apr 2013 13:23:01 +0200
>
> On 25.04.2013 01:46, Sage Weil wrote:
>> Hi everyone-
>>
>> We are down to a handful of urgent bugs (3!) and a cuttlefish release date
>> that is less than a week away. Thank you to everyone who has been
>> involved in coding, testing, and stabilizing this release. We are close!
>>
>> If you would like to test the current release candidate, your efforts
>> would be much appreciated! For deb systems, you can do
>>
>> wget -q -O- 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc' | sudo apt-key add -
>> echo deb http://gitbuilder.ceph.com/ceph-deb-$(lsb_release -sc)-x86_64-basic/ref/next $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
>>
>> For rpm users you can find packages at
>>
>> http://gitbuilder.ceph.com/ceph-rpm-centos6-x86_64-basic/ref/next/
>> http://gitbuilder.ceph.com/ceph-rpm-fc17-x86_64-basic/ref/next/
>> http://gitbuilder.ceph.com/ceph-rpm-fc18-x86_64-basic/ref/next/
>>
>> A draft of the release notes is up at
>>
>> http://ceph.com/docs/master/release-notes/#v0-61
>>
>> Let me know if I've missed anything!
>>
>> sage
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
Wido den Hollander
42on B.V.
Phone: +31 (0)20 700 9902
Skype: contact42on
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: cuttlefish countdown -- OSD doesn't get marked out
2013-04-25 12:56 ` Wido den Hollander
@ 2013-04-25 13:04 ` Martin Mailand
0 siblings, 0 replies; 16+ messages in thread
From: Martin Mailand @ 2013-04-25 13:04 UTC (permalink / raw)
To: Wido den Hollander, ceph-devel@vger.kernel.org, Sage Weil
Hi Wido,
I did not set the noosdout flag.
-martin
On 25.04.2013 14:56, Wido den Hollander wrote:
> Could you dump your osdmap? The first 10 lines would be interesting.
> There is a flag where you say "noosdout", could it be that the flag is set?
>
> Wido
epoch 206
fsid 13538f8a-a9b5-4f57-ad72-b0f9bec56a3e
created 2013-03-28 20:19:20.790878
modified 2013-04-25 13:09:55.825509
flags
pool 0 'data' rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins
pg_num 1600 pgp_num 1600 last_change 1 owner 0 crash_replay_interval 45
pool 1 'metadata' rep size 2 min_size 1 crush_ruleset 1 object_hash
rjenkins pg_num 1600 pgp_num 1600 last_change 1 owner 0
pool 2 'rbd' rep size 2 min_size 1 crush_ruleset 2 object_hash rjenkins
pg_num 1600 pgp_num 1600 last_change 77 owner 0
removed_snaps [1~1]
pool 3 'volumes' rep size 2 min_size 1 crush_ruleset 0 object_hash
rjenkins pg_num 128 pgp_num 128 last_change 199 owner 0
pool 4 'images' rep size 2 min_size 1 crush_ruleset 0 object_hash
rjenkins pg_num 128 pgp_num 128 last_change 204 owner 0
removed_snaps [1~1]
max_osd 24
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: cuttlefish countdown
2013-04-25 10:03 ` Kevin Decherf
@ 2013-04-25 16:05 ` Ian Colle
0 siblings, 0 replies; 16+ messages in thread
From: Ian Colle @ 2013-04-25 16:05 UTC (permalink / raw)
To: Kevin Decherf, Wido den Hollander; +Cc: Sage Weil, ceph-devel
On 4/25/13 4:03 AM, "Kevin Decherf" <kevin@kdecherf.com> wrote:
>Hi,
>
>On Thu, Apr 25, 2013 at 10:17:47AM +0200, Wido den Hollander wrote:
>> Hi,
>>
>> On 04/25/2013 01:46 AM, Sage Weil wrote:
>> >Hi everyone-
>> >
>> >We are down to a handful of urgent bugs (3!) and a cuttlefish release
>>date
>> >that is less than a week away. Thank you to everyone who has been
>> >involved in coding, testing, and stabilizing this release. We are
>>close!
>> >
>>
>> I'm trying to find the 3 urgent bugs for Cuttlefish, but I can't
>> seem to fetch them out of Redmine:
>> http://tracker.ceph.com/projects/1/issues?sort=position
>>
>> That doesn't show me any open urgent bugs? Trying to figure out
>> what's still open.
>
>http://tracker.ceph.com/projects/ceph/issues?utf8=%E2%9C%93&set_filter=1&f
>[]=status_id&op[status_id]=o&f[]=fixed_version_id&op[fixed_version_id]=%3D
>&v[fixed_version_id][]=138&f[]=&c[]=project&c[]=tracker&c[]=status&c[]=pri
>ority&c[]=subject&c[]=assigned_to&c[]=updated_on&c[]=category&c[]=fixed_ve
>rsion&c[]=story_points&c[]=cf_3&c[]=cf_4&group_by=
>
>That's what you are looking for?
Try the Bug Queue filter:
http://tracker.ceph.com/projects/ceph/issues?query_id=27
>
>--
>Kevin Decherf - @Kdecherf
>GPG C610 FE73 E706 F968 612B E4B2 108A BD75 A81E 6E2F
>http://kdecherf.com
>--
>To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: cuttlefish countdown
2013-04-25 8:17 ` Wido den Hollander
2013-04-25 10:03 ` Kevin Decherf
@ 2013-04-25 16:13 ` Sage Weil
1 sibling, 0 replies; 16+ messages in thread
From: Sage Weil @ 2013-04-25 16:13 UTC (permalink / raw)
To: Wido den Hollander; +Cc: ceph-devel
http://tracker.ceph.com/projects/ceph/issues?query_id=27
It's the 'Bug queue' link on the lower right under 'custom queries'. Now
there are 6 (ignoring pending backport)... :(
s
On Thu, 25 Apr 2013, Wido den Hollander wrote:
> Hi,
>
> On 04/25/2013 01:46 AM, Sage Weil wrote:
> > Hi everyone-
> >
> > We are down to a handful of urgent bugs (3!) and a cuttlefish release date
> > that is less than a week away. Thank you to everyone who has been
> > involved in coding, testing, and stabilizing this release. We are close!
> >
>
> I'm trying to find the 3 urgent bugs for Cuttlefish, but I can't seem to fetch
> them out of Redmine: http://tracker.ceph.com/projects/1/issues?sort=position
>
> That doesn't show me any open urgent bugs? Trying to figure out what's still
> open.
>
> Wido
>
> > If you would like to test the current release candidate, your efforts
> > would be much appreciated! For deb systems, you can do
> >
> > wget -q -O-
> > 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc' | sudo
> > apt-key add -
> > echo deb http://gitbuilder.ceph.com/ceph-deb-$(lsb_release
> > -sc)-x86_64-basic/ref/next $(lsb_release -sc) main | sudo tee
> > /etc/apt/sources.list.d/ceph.list
> >
> > For rpm users you can find packages at
> >
> > http://gitbuilder.ceph.com/ceph-rpm-centos6-x86_64-basic/ref/next/
> > http://gitbuilder.ceph.com/ceph-rpm-fc17-x86_64-basic/ref/next/
> > http://gitbuilder.ceph.com/ceph-rpm-fc18-x86_64-basic/ref/next/
> >
> > A draft of the release notes is up at
> >
> > http://ceph.com/docs/master/release-notes/#v0-61
> >
> > Let me know if I've missed anything!
> >
> > sage
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
>
>
> --
> Wido den Hollander
> 42on B.V.
>
> Phone: +31 (0)20 700 9902
> Skype: contact42on
>
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: cuttlefish countdown -- OSD doesn't get marked out
2013-04-25 12:07 ` cuttlefish countdown -- OSD doesn't get marked out Martin Mailand
2013-04-25 12:56 ` Wido den Hollander
@ 2013-04-25 16:17 ` Sage Weil
[not found] ` <alpine.DEB.2.00.1304250916430.4411-vIokxiIdD2AQNTJnQDzGJqxOck334EZe@public.gmane.org>
1 sibling, 1 reply; 16+ messages in thread
From: Sage Weil @ 2013-04-25 16:17 UTC (permalink / raw)
To: Martin Mailand; +Cc: ceph-devel, ceph-users
On Thu, 25 Apr 2013, Martin Mailand wrote:
> Hi,
>
> if I shutdown an OSD, the OSD gets marked down after 20 seconds, after
> 300 seconds the osd should get marked out, an the cluster should resync.
> But that doesn't happened, the OSD stays in the status down/in forever,
> therefore the cluster stays forever degraded.
> I can reproduce it with a new installed cluster.
>
> If I manually set the osd out (ceph osd out 1), the cluster resync
> starts immediately.
>
> I think thats a release critical bug, because the cluster health is not
> automatically recovered.
What is the output from 'ceph osd tree' and the contents of your
[mon*] sections of ceph.conf?
Thanks!
sage
>
> And I reported this behavior a while ago
> http://article.gmane.org/gmane.comp.file-systems.ceph.user/603/
>
> -martin
>
>
> Log:
>
>
> root@store1:~# ceph -s
> health HEALTH_OK
> monmap e1: 3 mons at
> {a=192.168.195.31:6789/0,b=192.168.195.33:6789/0,c=192.168.195.35:6789/0},
> election epoch 82, quorum 0,1,2 a,b,c
> osdmap e204: 24 osds: 24 up, 24 in
> pgmap v106709: 5056 pgs: 5056 active+clean; 526 GB data, 1068 GB
> used, 173 TB / 174 TB avail
> mdsmap e1: 0/0/1 up
>
> root@store1:~# ceph --version
> ceph version 0.60 (f26f7a39021dbf440c28d6375222e21c94fe8e5c)
> root@store1:~# /etc/init.d/ceph stop osd.1
> === osd.1 ===
> Stopping Ceph osd.1 on store1...bash: warning: setlocale: LC_ALL: cannot
> change locale (en_GB.utf8)
> kill 5492...done
> root@store1:~# ceph -s
> health HEALTH_OK
> monmap e1: 3 mons at
> {a=192.168.195.31:6789/0,b=192.168.195.33:6789/0,c=192.168.195.35:6789/0},
> election epoch 82, quorum 0,1,2 a,b,c
> osdmap e204: 24 osds: 24 up, 24 in
> pgmap v106709: 5056 pgs: 5056 active+clean; 526 GB data, 1068 GB
> used, 173 TB / 174 TB avail
> mdsmap e1: 0/0/1 up
>
> root@store1:~# date -R
> Thu, 25 Apr 2013 13:09:54 +0200
>
>
>
> root@store1:~# ceph -s && date -R
> health HEALTH_WARN 423 pgs degraded; 423 pgs stuck unclean; recovery
> 10999/269486 degraded (4.081%); 1/24 in osds are down
> monmap e1: 3 mons at
> {a=192.168.195.31:6789/0,b=192.168.195.33:6789/0,c=192.168.195.35:6789/0},
> election epoch 82, quorum 0,1,2 a,b,c
> osdmap e206: 24 osds: 23 up, 24 in
> pgmap v106715: 5056 pgs: 4633 active+clean, 423 active+degraded; 526
> GB data, 1068 GB used, 173 TB / 174 TB avail; 10999/269486 degraded (4.081%)
> mdsmap e1: 0/0/1 up
>
> Thu, 25 Apr 2013 13:10:14 +0200
>
>
> root@store1:~# ceph -s && date -R
> health HEALTH_WARN 423 pgs degraded; 423 pgs stuck unclean; recovery
> 10999/269486 degraded (4.081%); 1/24 in osds are down
> monmap e1: 3 mons at
> {a=192.168.195.31:6789/0,b=192.168.195.33:6789/0,c=192.168.195.35:6789/0},
> election epoch 82, quorum 0,1,2 a,b,c
> osdmap e206: 24 osds: 23 up, 24 in
> pgmap v106719: 5056 pgs: 4633 active+clean, 423 active+degraded; 526
> GB data, 1068 GB used, 173 TB / 174 TB avail; 10999/269486 degraded (4.081%)
> mdsmap e1: 0/0/1 up
>
> Thu, 25 Apr 2013 13:23:01 +0200
>
> On 25.04.2013 01:46, Sage Weil wrote:
> > Hi everyone-
> >
> > We are down to a handful of urgent bugs (3!) and a cuttlefish release date
> > that is less than a week away. Thank you to everyone who has been
> > involved in coding, testing, and stabilizing this release. We are close!
> >
> > If you would like to test the current release candidate, your efforts
> > would be much appreciated! For deb systems, you can do
> >
> > wget -q -O- 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc' | sudo apt-key add -
> > echo deb http://gitbuilder.ceph.com/ceph-deb-$(lsb_release -sc)-x86_64-basic/ref/next $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
> >
> > For rpm users you can find packages at
> >
> > http://gitbuilder.ceph.com/ceph-rpm-centos6-x86_64-basic/ref/next/
> > http://gitbuilder.ceph.com/ceph-rpm-fc17-x86_64-basic/ref/next/
> > http://gitbuilder.ceph.com/ceph-rpm-fc18-x86_64-basic/ref/next/
> >
> > A draft of the release notes is up at
> >
> > http://ceph.com/docs/master/release-notes/#v0-61
> >
> > Let me know if I've missed anything!
> >
> > sage
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: cuttlefish countdown -- OSD doesn't get marked out
[not found] ` <alpine.DEB.2.00.1304250916430.4411-vIokxiIdD2AQNTJnQDzGJqxOck334EZe@public.gmane.org>
@ 2013-04-25 16:38 ` Martin Mailand
[not found] ` <51795BE9.60601-64xna6+0MClWk0Htik3J/w@public.gmane.org>
2013-04-26 13:55 ` Mike Dawson
1 sibling, 1 reply; 16+ messages in thread
From: Martin Mailand @ 2013-04-25 16:38 UTC (permalink / raw)
To: Sage Weil; +Cc: ceph-devel-u79uwXL29TY76Z2rM5mHXA, ceph-users-Qp0mS5GaXlQ
Hi Sage,
On 25.04.2013 18:17, Sage Weil wrote:
> What is the output from 'ceph osd tree' and the contents of your
> [mon*] sections of ceph.conf?
>
> Thanks!
> sage
root@store1:~# ceph osd tree
# id weight type name up/down reweight
-1 24 root default
-3 24 rack unknownrack
-2 4 host store1
0 1 osd.0 up 1
1 1 osd.1 down 1
2 1 osd.2 up 1
3 1 osd.3 up 1
-4 4 host store3
10 1 osd.10 up 1
11 1 osd.11 up 1
8 1 osd.8 up 1
9 1 osd.9 up 1
-5 4 host store4
12 1 osd.12 up 1
13 1 osd.13 up 1
14 1 osd.14 up 1
15 1 osd.15 up 1
-6 4 host store5
16 1 osd.16 up 1
17 1 osd.17 up 1
18 1 osd.18 up 1
19 1 osd.19 up 1
-7 4 host store6
20 1 osd.20 up 1
21 1 osd.21 up 1
22 1 osd.22 up 1
23 1 osd.23 up 1
-8 4 host store2
4 1 osd.4 up 1
5 1 osd.5 up 1
6 1 osd.6 up 1
7 1 osd.7 up 1
[global]
auth cluster requierd = none
auth service required = none
auth client required = none
# log file = ""
log_max_recent=100
log_max_new=100
[mon]
mon data = /data/mon.$id
[mon.a]
mon host = store1
mon addr = 192.168.195.31:6789
[mon.b]
mon host = store3
mon addr = 192.168.195.33:6789
[mon.c]
mon host = store5
mon addr = 192.168.195.35:6789
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: cuttlefish countdown -- OSD doesn't get marked out
[not found] ` <51795BE9.60601-64xna6+0MClWk0Htik3J/w@public.gmane.org>
@ 2013-04-26 0:22 ` David Zafman
2013-04-26 8:50 ` [ceph-users] " Martin Mailand
0 siblings, 1 reply; 16+ messages in thread
From: David Zafman @ 2013-04-26 0:22 UTC (permalink / raw)
To: Martin Mailand; +Cc: ceph-devel-u79uwXL29TY76Z2rM5mHXA, ceph-users-Qp0mS5GaXlQ
I filed tracker bug 4822 and have wip-4822 with a fix. My manual testing shows that it works. I'm building a teuthology test.
Given your osd tree has a single rack it should always mark OSDs down after 5 minutes by default.
David Zafman
Senior Developer
http://www.inktank.com
On Apr 25, 2013, at 9:38 AM, Martin Mailand <martin-64xna6+0MClWk0Htik3J/w@public.gmane.org> wrote:
> Hi Sage,
>
> On 25.04.2013 18:17, Sage Weil wrote:
>> What is the output from 'ceph osd tree' and the contents of your
>> [mon*] sections of ceph.conf?
>>
>> Thanks!
>> sage
>
>
> root@store1:~# ceph osd tree
>
> # id weight type name up/down reweight
> -1 24 root default
> -3 24 rack unknownrack
> -2 4 host store1
> 0 1 osd.0 up 1
> 1 1 osd.1 down 1
> 2 1 osd.2 up 1
> 3 1 osd.3 up 1
> -4 4 host store3
> 10 1 osd.10 up 1
> 11 1 osd.11 up 1
> 8 1 osd.8 up 1
> 9 1 osd.9 up 1
> -5 4 host store4
> 12 1 osd.12 up 1
> 13 1 osd.13 up 1
> 14 1 osd.14 up 1
> 15 1 osd.15 up 1
> -6 4 host store5
> 16 1 osd.16 up 1
> 17 1 osd.17 up 1
> 18 1 osd.18 up 1
> 19 1 osd.19 up 1
> -7 4 host store6
> 20 1 osd.20 up 1
> 21 1 osd.21 up 1
> 22 1 osd.22 up 1
> 23 1 osd.23 up 1
> -8 4 host store2
> 4 1 osd.4 up 1
> 5 1 osd.5 up 1
> 6 1 osd.6 up 1
> 7 1 osd.7 up 1
>
>
>
> [global]
> auth cluster requierd = none
> auth service required = none
> auth client required = none
> # log file = ""
> log_max_recent=100
> log_max_new=100
>
> [mon]
> mon data = /data/mon.$id
> [mon.a]
> mon host = store1
> mon addr = 192.168.195.31:6789
> [mon.b]
> mon host = store3
> mon addr = 192.168.195.33:6789
> [mon.c]
> mon host = store5
> mon addr = 192.168.195.35:6789
> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [ceph-users] cuttlefish countdown -- OSD doesn't get marked out
2013-04-26 0:22 ` David Zafman
@ 2013-04-26 8:50 ` Martin Mailand
2013-04-26 13:44 ` Mike Dawson
2013-04-26 17:02 ` David Zafman
0 siblings, 2 replies; 16+ messages in thread
From: Martin Mailand @ 2013-04-26 8:50 UTC (permalink / raw)
To: David Zafman; +Cc: ceph-devel
Hi David,
did you test it with more than one rack as well? In my first problem I
used two racks, with a custom crushmap, so that the replicas are in the
two racks (replicationlevel = 2). Than I took one osd down, and expected
that the remaining osds in this rack would get the now missing replicas
from the osd of the other rack.
But nothing happened, the cluster stayed degraded.
-martin
On 26.04.2013 02:22, David Zafman wrote:
>
> I filed tracker bug 4822 and have wip-4822 with a fix. My manual testing shows that it works. I'm building a teuthology test.
>
> Given your osd tree has a single rack it should always mark OSDs down after 5 minutes by default.
>
> David Zafman
> Senior Developer
> http://www.inktank.com
>
>
>
>
> On Apr 25, 2013, at 9:38 AM, Martin Mailand <martin@tuxadero.com> wrote:
>
>> Hi Sage,
>>
>> On 25.04.2013 18:17, Sage Weil wrote:
>>> What is the output from 'ceph osd tree' and the contents of your
>>> [mon*] sections of ceph.conf?
>>>
>>> Thanks!
>>> sage
>>
>>
>> root@store1:~# ceph osd tree
>>
>> # id weight type name up/down reweight
>> -1 24 root default
>> -3 24 rack unknownrack
>> -2 4 host store1
>> 0 1 osd.0 up 1
>> 1 1 osd.1 down 1
>> 2 1 osd.2 up 1
>> 3 1 osd.3 up 1
>> -4 4 host store3
>> 10 1 osd.10 up 1
>> 11 1 osd.11 up 1
>> 8 1 osd.8 up 1
>> 9 1 osd.9 up 1
>> -5 4 host store4
>> 12 1 osd.12 up 1
>> 13 1 osd.13 up 1
>> 14 1 osd.14 up 1
>> 15 1 osd.15 up 1
>> -6 4 host store5
>> 16 1 osd.16 up 1
>> 17 1 osd.17 up 1
>> 18 1 osd.18 up 1
>> 19 1 osd.19 up 1
>> -7 4 host store6
>> 20 1 osd.20 up 1
>> 21 1 osd.21 up 1
>> 22 1 osd.22 up 1
>> 23 1 osd.23 up 1
>> -8 4 host store2
>> 4 1 osd.4 up 1
>> 5 1 osd.5 up 1
>> 6 1 osd.6 up 1
>> 7 1 osd.7 up 1
>>
>>
>>
>> [global]
>> auth cluster requierd = none
>> auth service required = none
>> auth client required = none
>> # log file = ""
>> log_max_recent=100
>> log_max_new=100
>>
>> [mon]
>> mon data = /data/mon.$id
>> [mon.a]
>> mon host = store1
>> mon addr = 192.168.195.31:6789
>> [mon.b]
>> mon host = store3
>> mon addr = 192.168.195.33:6789
>> [mon.c]
>> mon host = store5
>> mon addr = 192.168.195.35:6789
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [ceph-users] cuttlefish countdown -- OSD doesn't get marked out
2013-04-26 8:50 ` [ceph-users] " Martin Mailand
@ 2013-04-26 13:44 ` Mike Dawson
2013-04-26 18:18 ` David Zafman
2013-04-26 17:02 ` David Zafman
1 sibling, 1 reply; 16+ messages in thread
From: Mike Dawson @ 2013-04-26 13:44 UTC (permalink / raw)
To: Martin Mailand, David Zafman; +Cc: ceph-devel
David / Martin,
I can confirm this issue. At present I am running monitors only with
100% of my OSD processes shutdown down. For the past couple hours, Ceph
has reported:
osdmap e1323: 66 osds: 19 up, 66 in
I can mark them down manually using
ceph osd down 0
as expected, but they never get marked down automatically. Like Martin,
I also have a custom crushmap, but this cluster is operating with a
single rack. I'll be happy to provide any documentation / configs / logs
you would like.
I am currently running ceph version 0.60-666-ga5cade1
(a5cade1fe7338602fb2bbfa867433d825f337c87) from gitbuilder.
- Mike
On 4/26/2013 4:50 AM, Martin Mailand wrote:
> Hi David,
>
> did you test it with more than one rack as well? In my first problem I
> used two racks, with a custom crushmap, so that the replicas are in the
> two racks (replicationlevel = 2). Than I took one osd down, and expected
> that the remaining osds in this rack would get the now missing replicas
> from the osd of the other rack.
> But nothing happened, the cluster stayed degraded.
>
> -martin
>
>
> On 26.04.2013 02:22, David Zafman wrote:
>>
>> I filed tracker bug 4822 and have wip-4822 with a fix. My manual testing shows that it works. I'm building a teuthology test.
>>
>> Given your osd tree has a single rack it should always mark OSDs down after 5 minutes by default.
>>
>> David Zafman
>> Senior Developer
>> http://www.inktank.com
>>
>>
>>
>>
>> On Apr 25, 2013, at 9:38 AM, Martin Mailand <martin@tuxadero.com> wrote:
>>
>>> Hi Sage,
>>>
>>> On 25.04.2013 18:17, Sage Weil wrote:
>>>> What is the output from 'ceph osd tree' and the contents of your
>>>> [mon*] sections of ceph.conf?
>>>>
>>>> Thanks!
>>>> sage
>>>
>>>
>>> root@store1:~# ceph osd tree
>>>
>>> # id weight type name up/down reweight
>>> -1 24 root default
>>> -3 24 rack unknownrack
>>> -2 4 host store1
>>> 0 1 osd.0 up 1
>>> 1 1 osd.1 down 1
>>> 2 1 osd.2 up 1
>>> 3 1 osd.3 up 1
>>> -4 4 host store3
>>> 10 1 osd.10 up 1
>>> 11 1 osd.11 up 1
>>> 8 1 osd.8 up 1
>>> 9 1 osd.9 up 1
>>> -5 4 host store4
>>> 12 1 osd.12 up 1
>>> 13 1 osd.13 up 1
>>> 14 1 osd.14 up 1
>>> 15 1 osd.15 up 1
>>> -6 4 host store5
>>> 16 1 osd.16 up 1
>>> 17 1 osd.17 up 1
>>> 18 1 osd.18 up 1
>>> 19 1 osd.19 up 1
>>> -7 4 host store6
>>> 20 1 osd.20 up 1
>>> 21 1 osd.21 up 1
>>> 22 1 osd.22 up 1
>>> 23 1 osd.23 up 1
>>> -8 4 host store2
>>> 4 1 osd.4 up 1
>>> 5 1 osd.5 up 1
>>> 6 1 osd.6 up 1
>>> 7 1 osd.7 up 1
>>>
>>>
>>>
>>> [global]
>>> auth cluster requierd = none
>>> auth service required = none
>>> auth client required = none
>>> # log file = ""
>>> log_max_recent=100
>>> log_max_new=100
>>>
>>> [mon]
>>> mon data = /data/mon.$id
>>> [mon.a]
>>> mon host = store1
>>> mon addr = 192.168.195.31:6789
>>> [mon.b]
>>> mon host = store3
>>> mon addr = 192.168.195.33:6789
>>> [mon.c]
>>> mon host = store5
>>> mon addr = 192.168.195.35:6789
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: cuttlefish countdown -- OSD doesn't get marked out
[not found] ` <alpine.DEB.2.00.1304250916430.4411-vIokxiIdD2AQNTJnQDzGJqxOck334EZe@public.gmane.org>
2013-04-25 16:38 ` Martin Mailand
@ 2013-04-26 13:55 ` Mike Dawson
1 sibling, 0 replies; 16+ messages in thread
From: Mike Dawson @ 2013-04-26 13:55 UTC (permalink / raw)
To: Sage Weil; +Cc: ceph-devel-u79uwXL29TY76Z2rM5mHXA, ceph-users-Qp0mS5GaXlQ
Sage,
I confirm this issue. The requested info is listed below.
*Note that due to the pre-Cuttlefish monitor sync issues, this
deployment has been running three monitors (mon.b and mon.c working
properly in quorum. mon.a stuck forever synchronizing).
For the past two hours, no OSD processes have been running on any host,
yet some OSDs are still marked as up.
http://www.gammacode.com/upload/ceph-osd-tree
The mon* sections of ceph.conf are:
[mon]
debug mon = 20
debug paxos = 20
debug ms = 1
[mon.a]
host = node2
mon addr = 10.1.0.3:6789
[mon.b]
host = node26
mon addr = 10.1.0.67:6789
[mon.c]
host = node49
mon addr = 10.1.0.130:6789
root@controller1:~# ceph -s
health HEALTH_WARN 43 pgs degraded; 13308 pgs peering; 27932 pgs
stale; 13308 pgs stuck inactive; 27932 pgs stuck stale; 13582 pgs stuck
unclean; recovery 7264/7986546 degraded (0.091%); 47/66 in osds are
down; 1 mons down, quorum 1,2 b,c
monmap e1: 3 mons at
{a=10.1.0.3:6789/0,b=10.1.0.67:6789/0,c=10.1.0.130:6789/0}, election
epoch 1428, quorum 1,2 b,c
osdmap e1323: 66 osds: 19 up, 66 in
pgmap v427324: 28864 pgs: 257 active+clean, 231 stale+active, 15025
stale+active+clean, 675 peering, 12633 stale+peering, 43
stale+active+degraded; 448 GB data, 1402 GB used, 178 TB / 180 TB avail;
7264/7986546 degraded (0.091%)
mdsmap e1: 0/0/1 up
For reference, this is ceph version 0.60-666-ga5cade1
(a5cade1fe7338602fb2bbfa867433d825f337c87) from gitbuilder.
Thanks,
Mike
On 4/25/2013 12:17 PM, Sage Weil wrote:
> On Thu, 25 Apr 2013, Martin Mailand wrote:
>> Hi,
>>
>> if I shutdown an OSD, the OSD gets marked down after 20 seconds, after
>> 300 seconds the osd should get marked out, an the cluster should resync.
>> But that doesn't happened, the OSD stays in the status down/in forever,
>> therefore the cluster stays forever degraded.
>> I can reproduce it with a new installed cluster.
>>
>> If I manually set the osd out (ceph osd out 1), the cluster resync
>> starts immediately.
>>
>> I think thats a release critical bug, because the cluster health is not
>> automatically recovered.
>
> What is the output from 'ceph osd tree' and the contents of your
> [mon*] sections of ceph.conf?
>
> Thanks!
> sage
>
>
>>
>> And I reported this behavior a while ago
>> http://article.gmane.org/gmane.comp.file-systems.ceph.user/603/
>>
>> -martin
>>
>>
>> Log:
>>
>>
>> root@store1:~# ceph -s
>> health HEALTH_OK
>> monmap e1: 3 mons at
>> {a=192.168.195.31:6789/0,b=192.168.195.33:6789/0,c=192.168.195.35:6789/0},
>> election epoch 82, quorum 0,1,2 a,b,c
>> osdmap e204: 24 osds: 24 up, 24 in
>> pgmap v106709: 5056 pgs: 5056 active+clean; 526 GB data, 1068 GB
>> used, 173 TB / 174 TB avail
>> mdsmap e1: 0/0/1 up
>>
>> root@store1:~# ceph --version
>> ceph version 0.60 (f26f7a39021dbf440c28d6375222e21c94fe8e5c)
>> root@store1:~# /etc/init.d/ceph stop osd.1
>> === osd.1 ===
>> Stopping Ceph osd.1 on store1...bash: warning: setlocale: LC_ALL: cannot
>> change locale (en_GB.utf8)
>> kill 5492...done
>> root@store1:~# ceph -s
>> health HEALTH_OK
>> monmap e1: 3 mons at
>> {a=192.168.195.31:6789/0,b=192.168.195.33:6789/0,c=192.168.195.35:6789/0},
>> election epoch 82, quorum 0,1,2 a,b,c
>> osdmap e204: 24 osds: 24 up, 24 in
>> pgmap v106709: 5056 pgs: 5056 active+clean; 526 GB data, 1068 GB
>> used, 173 TB / 174 TB avail
>> mdsmap e1: 0/0/1 up
>>
>> root@store1:~# date -R
>> Thu, 25 Apr 2013 13:09:54 +0200
>>
>>
>>
>> root@store1:~# ceph -s && date -R
>> health HEALTH_WARN 423 pgs degraded; 423 pgs stuck unclean; recovery
>> 10999/269486 degraded (4.081%); 1/24 in osds are down
>> monmap e1: 3 mons at
>> {a=192.168.195.31:6789/0,b=192.168.195.33:6789/0,c=192.168.195.35:6789/0},
>> election epoch 82, quorum 0,1,2 a,b,c
>> osdmap e206: 24 osds: 23 up, 24 in
>> pgmap v106715: 5056 pgs: 4633 active+clean, 423 active+degraded; 526
>> GB data, 1068 GB used, 173 TB / 174 TB avail; 10999/269486 degraded (4.081%)
>> mdsmap e1: 0/0/1 up
>>
>> Thu, 25 Apr 2013 13:10:14 +0200
>>
>>
>> root@store1:~# ceph -s && date -R
>> health HEALTH_WARN 423 pgs degraded; 423 pgs stuck unclean; recovery
>> 10999/269486 degraded (4.081%); 1/24 in osds are down
>> monmap e1: 3 mons at
>> {a=192.168.195.31:6789/0,b=192.168.195.33:6789/0,c=192.168.195.35:6789/0},
>> election epoch 82, quorum 0,1,2 a,b,c
>> osdmap e206: 24 osds: 23 up, 24 in
>> pgmap v106719: 5056 pgs: 4633 active+clean, 423 active+degraded; 526
>> GB data, 1068 GB used, 173 TB / 174 TB avail; 10999/269486 degraded (4.081%)
>> mdsmap e1: 0/0/1 up
>>
>> Thu, 25 Apr 2013 13:23:01 +0200
>>
>> On 25.04.2013 01:46, Sage Weil wrote:
>>> Hi everyone-
>>>
>>> We are down to a handful of urgent bugs (3!) and a cuttlefish release date
>>> that is less than a week away. Thank you to everyone who has been
>>> involved in coding, testing, and stabilizing this release. We are close!
>>>
>>> If you would like to test the current release candidate, your efforts
>>> would be much appreciated! For deb systems, you can do
>>>
>>> wget -q -O- 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc' | sudo apt-key add -
>>> echo deb http://gitbuilder.ceph.com/ceph-deb-$(lsb_release -sc)-x86_64-basic/ref/next $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
>>>
>>> For rpm users you can find packages at
>>>
>>> http://gitbuilder.ceph.com/ceph-rpm-centos6-x86_64-basic/ref/next/
>>> http://gitbuilder.ceph.com/ceph-rpm-fc17-x86_64-basic/ref/next/
>>> http://gitbuilder.ceph.com/ceph-rpm-fc18-x86_64-basic/ref/next/
>>>
>>> A draft of the release notes is up at
>>>
>>> http://ceph.com/docs/master/release-notes/#v0-61
>>>
>>> Let me know if I've missed anything!
>>>
>>> sage
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [ceph-users] cuttlefish countdown -- OSD doesn't get marked out
2013-04-26 8:50 ` [ceph-users] " Martin Mailand
2013-04-26 13:44 ` Mike Dawson
@ 2013-04-26 17:02 ` David Zafman
1 sibling, 0 replies; 16+ messages in thread
From: David Zafman @ 2013-04-26 17:02 UTC (permalink / raw)
To: Martin Mailand; +Cc: ceph-devel
The behavior you are seeing is exactly what would be expected if OSDs are not being marked out. The testing of my fix showed that if a portion of a rack's OSDs go down they will be marked out after the configured amount of time (5 min by default). Once down OSDs are out the remaining OSDs take responsibility for holding the data assigned to that rack.
Though I didn't look at the data movement, I'm confident that it will work. You can simply mark your OSDs out manually to verify that missing replicas are replaced.
David Zafman
Senior Developer
http://www.inktank.com
On Apr 26, 2013, at 1:50 AM, Martin Mailand <martin@tuxadero.com> wrote:
> Hi David,
>
> did you test it with more than one rack as well? In my first problem I
> used two racks, with a custom crushmap, so that the replicas are in the
> two racks (replicationlevel = 2). Than I took one osd down, and expected
> that the remaining osds in this rack would get the now missing replicas
> from the osd of the other rack.
> But nothing happened, the cluster stayed degraded.
>
> -martin
>
>
> On 26.04.2013 02:22, David Zafman wrote:
>>
>> I filed tracker bug 4822 and have wip-4822 with a fix. My manual testing shows that it works. I'm building a teuthology test.
>>
>> Given your osd tree has a single rack it should always mark OSDs down after 5 minutes by default.
>>
>> David Zafman
>> Senior Developer
>> http://www.inktank.com
>>
>>
>>
>>
>> On Apr 25, 2013, at 9:38 AM, Martin Mailand <martin@tuxadero.com> wrote:
>>
>>> Hi Sage,
>>>
>>> On 25.04.2013 18:17, Sage Weil wrote:
>>>> What is the output from 'ceph osd tree' and the contents of your
>>>> [mon*] sections of ceph.conf?
>>>>
>>>> Thanks!
>>>> sage
>>>
>>>
>>> root@store1:~# ceph osd tree
>>>
>>> # id weight type name up/down reweight
>>> -1 24 root default
>>> -3 24 rack unknownrack
>>> -2 4 host store1
>>> 0 1 osd.0 up 1
>>> 1 1 osd.1 down 1
>>> 2 1 osd.2 up 1
>>> 3 1 osd.3 up 1
>>> -4 4 host store3
>>> 10 1 osd.10 up 1
>>> 11 1 osd.11 up 1
>>> 8 1 osd.8 up 1
>>> 9 1 osd.9 up 1
>>> -5 4 host store4
>>> 12 1 osd.12 up 1
>>> 13 1 osd.13 up 1
>>> 14 1 osd.14 up 1
>>> 15 1 osd.15 up 1
>>> -6 4 host store5
>>> 16 1 osd.16 up 1
>>> 17 1 osd.17 up 1
>>> 18 1 osd.18 up 1
>>> 19 1 osd.19 up 1
>>> -7 4 host store6
>>> 20 1 osd.20 up 1
>>> 21 1 osd.21 up 1
>>> 22 1 osd.22 up 1
>>> 23 1 osd.23 up 1
>>> -8 4 host store2
>>> 4 1 osd.4 up 1
>>> 5 1 osd.5 up 1
>>> 6 1 osd.6 up 1
>>> 7 1 osd.7 up 1
>>>
>>>
>>>
>>> [global]
>>> auth cluster requierd = none
>>> auth service required = none
>>> auth client required = none
>>> # log file = ""
>>> log_max_recent=100
>>> log_max_new=100
>>>
>>> [mon]
>>> mon data = /data/mon.$id
>>> [mon.a]
>>> mon host = store1
>>> mon addr = 192.168.195.31:6789
>>> [mon.b]
>>> mon host = store3
>>> mon addr = 192.168.195.33:6789
>>> [mon.c]
>>> mon host = store5
>>> mon addr = 192.168.195.35:6789
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [ceph-users] cuttlefish countdown -- OSD doesn't get marked out
2013-04-26 13:44 ` Mike Dawson
@ 2013-04-26 18:18 ` David Zafman
0 siblings, 0 replies; 16+ messages in thread
From: David Zafman @ 2013-04-26 18:18 UTC (permalink / raw)
To: Mike Dawson; +Cc: Martin Mailand, ceph-devel
Mike / Martin,
The OSD down behavior Mike is seeing is different. You should be seeing messages like this in your leader's monitor log:
can_mark_down current up_ratio 0.166667 < min 0.3, will not mark osd.2 down
To dampen certain kinds of cascading failures, we are deliberately restricting automatically marking < 30% of OSDs down.
As far as Martin is concerned his osd tree shows a single rack, but said that his crush rules are supposed to put a replica on each of 2 racks. I don't remember seeing his crush rules in any of the e-mails, but even so he only has "unknownrack" with id -3 defined.
David Zafman
Senior Developer
http://www.inktank.com
On Apr 26, 2013, at 6:44 AM, Mike Dawson <mike.dawson@scholarstack.com> wrote:
> David / Martin,
>
> I can confirm this issue. At present I am running monitors only with 100% of my OSD processes shutdown down. For the past couple hours, Ceph has reported:
>
> osdmap e1323: 66 osds: 19 up, 66 in
>
> I can mark them down manually using
>
> ceph osd down 0
>
> as expected, but they never get marked down automatically. Like Martin, I also have a custom crushmap, but this cluster is operating with a single rack. I'll be happy to provide any documentation / configs / logs you would like.
>
> I am currently running ceph version 0.60-666-ga5cade1 (a5cade1fe7338602fb2bbfa867433d825f337c87) from gitbuilder.
>
> - Mike
>
> On 4/26/2013 4:50 AM, Martin Mailand wrote:
>> Hi David,
>>
>> did you test it with more than one rack as well? In my first problem I
>> used two racks, with a custom crushmap, so that the replicas are in the
>> two racks (replicationlevel = 2). Than I took one osd down, and expected
>> that the remaining osds in this rack would get the now missing replicas
>> from the osd of the other rack.
>> But nothing happened, the cluster stayed degraded.
>>
>> -martin
>>
>>
>> On 26.04.2013 02:22, David Zafman wrote:
>>>
>>> I filed tracker bug 4822 and have wip-4822 with a fix. My manual testing shows that it works. I'm building a teuthology test.
>>>
>>> Given your osd tree has a single rack it should always mark OSDs down after 5 minutes by default.
>>>
>>> David Zafman
>>> Senior Developer
>>> http://www.inktank.com
>>>
>>>
>>>
>>>
>>> On Apr 25, 2013, at 9:38 AM, Martin Mailand <martin@tuxadero.com> wrote:
>>>
>>>> Hi Sage,
>>>>
>>>> On 25.04.2013 18:17, Sage Weil wrote:
>>>>> What is the output from 'ceph osd tree' and the contents of your
>>>>> [mon*] sections of ceph.conf?
>>>>>
>>>>> Thanks!
>>>>> sage
>>>>
>>>>
>>>> root@store1:~# ceph osd tree
>>>>
>>>> # id weight type name up/down reweight
>>>> -1 24 root default
>>>> -3 24 rack unknownrack
>>>> -2 4 host store1
>>>> 0 1 osd.0 up 1
>>>> 1 1 osd.1 down 1
>>>> 2 1 osd.2 up 1
>>>> 3 1 osd.3 up 1
>>>> -4 4 host store3
>>>> 10 1 osd.10 up 1
>>>> 11 1 osd.11 up 1
>>>> 8 1 osd.8 up 1
>>>> 9 1 osd.9 up 1
>>>> -5 4 host store4
>>>> 12 1 osd.12 up 1
>>>> 13 1 osd.13 up 1
>>>> 14 1 osd.14 up 1
>>>> 15 1 osd.15 up 1
>>>> -6 4 host store5
>>>> 16 1 osd.16 up 1
>>>> 17 1 osd.17 up 1
>>>> 18 1 osd.18 up 1
>>>> 19 1 osd.19 up 1
>>>> -7 4 host store6
>>>> 20 1 osd.20 up 1
>>>> 21 1 osd.21 up 1
>>>> 22 1 osd.22 up 1
>>>> 23 1 osd.23 up 1
>>>> -8 4 host store2
>>>> 4 1 osd.4 up 1
>>>> 5 1 osd.5 up 1
>>>> 6 1 osd.6 up 1
>>>> 7 1 osd.7 up 1
>>>>
>>>>
>>>>
>>>> [global]
>>>> auth cluster requierd = none
>>>> auth service required = none
>>>> auth client required = none
>>>> # log file = ""
>>>> log_max_recent=100
>>>> log_max_new=100
>>>>
>>>> [mon]
>>>> mon data = /data/mon.$id
>>>> [mon.a]
>>>> mon host = store1
>>>> mon addr = 192.168.195.31:6789
>>>> [mon.b]
>>>> mon host = store3
>>>> mon addr = 192.168.195.33:6789
>>>> [mon.c]
>>>> mon host = store5
>>>> mon addr = 192.168.195.35:6789
>>>> _______________________________________________
>>>> ceph-users mailing list
>>>> ceph-users@lists.ceph.com
>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2013-04-26 18:18 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-04-24 23:46 cuttlefish countdown Sage Weil
2013-04-25 8:17 ` Wido den Hollander
2013-04-25 10:03 ` Kevin Decherf
2013-04-25 16:05 ` Ian Colle
2013-04-25 16:13 ` Sage Weil
2013-04-25 12:07 ` cuttlefish countdown -- OSD doesn't get marked out Martin Mailand
2013-04-25 12:56 ` Wido den Hollander
2013-04-25 13:04 ` Martin Mailand
2013-04-25 16:17 ` Sage Weil
[not found] ` <alpine.DEB.2.00.1304250916430.4411-vIokxiIdD2AQNTJnQDzGJqxOck334EZe@public.gmane.org>
2013-04-25 16:38 ` Martin Mailand
[not found] ` <51795BE9.60601-64xna6+0MClWk0Htik3J/w@public.gmane.org>
2013-04-26 0:22 ` David Zafman
2013-04-26 8:50 ` [ceph-users] " Martin Mailand
2013-04-26 13:44 ` Mike Dawson
2013-04-26 18:18 ` David Zafman
2013-04-26 17:02 ` David Zafman
2013-04-26 13:55 ` Mike Dawson
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.