* Available space shrinks to zero after upgrading cluster
@ 2012-01-09 11:23 Guido Winkelmann
2012-01-09 11:28 ` Guido Winkelmann
0 siblings, 1 reply; 4+ messages in thread
From: Guido Winkelmann @ 2012-01-09 11:23 UTC (permalink / raw)
To: ceph-devel
Hi,
I just upgraded my cluster to the current git version (from 0.39, using
https://github.com/NewDreamNetwork/ceph.git), and now ceph -s reports the
total available space in the the cluster as 0 and all of the clients using the
cluster are blocking:
# ceph -s
2012-01-09 12:12:39.433576 pg v313084: 396 pgs: 396 active+clean; 79166 MB
data, 0 KB used, 0 KB / 0 KB avail
2012-01-09 12:12:39.434269 mds e11: 1/1/1 up {0=alpha=up:replay}
2012-01-09 12:12:39.434289 osd e293: 6 osds: 6 up, 6 in
2012-01-09 12:12:39.434338 log 2012-01-09 12:12:26.600575 mon.0
10.3.1.33:6789/0 6 : [INF] osd.1 10.3.1.33:6804/22161 boot
2012-01-09 12:12:39.434389 mon e5: 3 mons at
{ceph1=10.3.1.33:6789/0,ceph2=10.3.1.34:6789/0,ceph3=10.3.1.35:6789/0}
I have a cluster with three machines, each with two OSDs and one mon.
Additionally, the first one also has a single mds.
I did the upgrade by cloning the latest git sources onto each machine, doing
the usual ./autgen.sh, ./configure, make && make install and then restarting
ceph on one machine after the other using /etc/init.d/ceph restart, starting
with the last one.
So... what's happening here and how do I get my cluster working again?
Guido
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Available space shrinks to zero after upgrading cluster
2012-01-09 11:23 Available space shrinks to zero after upgrading cluster Guido Winkelmann
@ 2012-01-09 11:28 ` Guido Winkelmann
2012-01-09 23:40 ` Samuel Just
0 siblings, 1 reply; 4+ messages in thread
From: Guido Winkelmann @ 2012-01-09 11:28 UTC (permalink / raw)
To: ceph-devel
Am Montag, 9. Januar 2012, 12:23:53 schrieb Guido Winkelmann:
> Hi,
>
> I just upgraded my cluster to the current git version (from 0.39, using
> https://github.com/NewDreamNetwork/ceph.git), and now ceph -s reports the
> total available space in the the cluster as 0 and all of the clients using
> the cluster are blocking:
>
> # ceph -s
> 2012-01-09 12:12:39.433576 pg v313084: 396 pgs: 396 active+clean; 79166
> MB data, 0 KB used, 0 KB / 0 KB avail
> 2012-01-09 12:12:39.434269 mds e11: 1/1/1 up {0=alpha=up:replay}
> 2012-01-09 12:12:39.434289 osd e293: 6 osds: 6 up, 6 in
> 2012-01-09 12:12:39.434338 log 2012-01-09 12:12:26.600575 mon.0
> 10.3.1.33:6789/0 6 : [INF] osd.1 10.3.1.33:6804/22161 boot
> 2012-01-09 12:12:39.434389 mon e5: 3 mons at
> {ceph1=10.3.1.33:6789/0,ceph2=10.3.1.34:6789/0,ceph3=10.3.1.35:6789/0}
>
> I have a cluster with three machines, each with two OSDs and one mon.
> Additionally, the first one also has a single mds.
>
> I did the upgrade by cloning the latest git sources onto each machine, doing
> the usual ./autgen.sh, ./configure, make && make install and then
> restarting ceph on one machine after the other using /etc/init.d/ceph
> restart, starting with the last one.
>
> So... what's happening here and how do I get my cluster working again?
Oh, hey, it just changed to this:
# ceph -s
2012-01-09 12:28:07.745570 pg v313092: 396 pgs: 232 active+clean, 106
active+clean+degraded, 13 active+clean+replay+degraded, 45
down+degraded+peering; 79166 MB data, 33829 MB used, 825 GB / 876 GB avail;
8792/42838 degraded (20.524%)
2012-01-09 12:28:07.746310 mds e11: 1/1/1 up {0=alpha=up:replay}
2012-01-09 12:28:07.746330 osd e296: 6 osds: 1 up, 6 in
2012-01-09 12:28:07.746379 log 2012-01-09 12:27:24.965150 mon.0
10.3.1.33:6789/0 7 : [INF] osd.4 10.3.1.35:6806/3705 boot
2012-01-09 12:28:07.746422 mon e1: 1 mons at {ceph1=10.3.1.33:6789/0}
2012-01-09 12:28:07.746431 mon e2: 2 mons at
{ceph1=10.3.1.33:6789/0,ceph2=10.3.1.34:6789/0}
2012-01-09 12:28:07.746440 mon e3: 1 mons at {ceph1=10.3.1.33:6789/0}
2012-01-09 12:28:07.746446 mon e4: 2 mons at
{ceph1=10.3.1.33:6789/0,ceph2=10.3.1.34:6789/0}
2012-01-09 12:28:07.746453 mon e5: 3 mons at
{ceph1=10.3.1.33:6789/0,ceph2=10.3.1.34:6789/0,ceph3=10.3.1.35:6789/0}
Guido
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Available space shrinks to zero after upgrading cluster
2012-01-09 11:28 ` Guido Winkelmann
@ 2012-01-09 23:40 ` Samuel Just
2012-01-11 15:10 ` Guido Winkelmann
0 siblings, 1 reply; 4+ messages in thread
From: Samuel Just @ 2012-01-09 23:40 UTC (permalink / raw)
To: Guido Winkelmann; +Cc: ceph-devel
After restarting the OSDs, does the available space make sense?
-Sam
On Mon, Jan 9, 2012 at 3:28 AM, Guido Winkelmann
<guido-ceph@thisisnotatest.de> wrote:
> Am Montag, 9. Januar 2012, 12:23:53 schrieb Guido Winkelmann:
>> Hi,
>>
>> I just upgraded my cluster to the current git version (from 0.39, using
>> https://github.com/NewDreamNetwork/ceph.git), and now ceph -s reports the
>> total available space in the the cluster as 0 and all of the clients using
>> the cluster are blocking:
>>
>> # ceph -s
>> 2012-01-09 12:12:39.433576 pg v313084: 396 pgs: 396 active+clean; 79166
>> MB data, 0 KB used, 0 KB / 0 KB avail
>> 2012-01-09 12:12:39.434269 mds e11: 1/1/1 up {0=alpha=up:replay}
>> 2012-01-09 12:12:39.434289 osd e293: 6 osds: 6 up, 6 in
>> 2012-01-09 12:12:39.434338 log 2012-01-09 12:12:26.600575 mon.0
>> 10.3.1.33:6789/0 6 : [INF] osd.1 10.3.1.33:6804/22161 boot
>> 2012-01-09 12:12:39.434389 mon e5: 3 mons at
>> {ceph1=10.3.1.33:6789/0,ceph2=10.3.1.34:6789/0,ceph3=10.3.1.35:6789/0}
>>
>> I have a cluster with three machines, each with two OSDs and one mon.
>> Additionally, the first one also has a single mds.
>>
>> I did the upgrade by cloning the latest git sources onto each machine, doing
>> the usual ./autgen.sh, ./configure, make && make install and then
>> restarting ceph on one machine after the other using /etc/init.d/ceph
>> restart, starting with the last one.
>>
>> So... what's happening here and how do I get my cluster working again?
>
> Oh, hey, it just changed to this:
>
> # ceph -s
> 2012-01-09 12:28:07.745570 pg v313092: 396 pgs: 232 active+clean, 106
> active+clean+degraded, 13 active+clean+replay+degraded, 45
> down+degraded+peering; 79166 MB data, 33829 MB used, 825 GB / 876 GB avail;
> 8792/42838 degraded (20.524%)
> 2012-01-09 12:28:07.746310 mds e11: 1/1/1 up {0=alpha=up:replay}
> 2012-01-09 12:28:07.746330 osd e296: 6 osds: 1 up, 6 in
> 2012-01-09 12:28:07.746379 log 2012-01-09 12:27:24.965150 mon.0
> 10.3.1.33:6789/0 7 : [INF] osd.4 10.3.1.35:6806/3705 boot
> 2012-01-09 12:28:07.746422 mon e1: 1 mons at {ceph1=10.3.1.33:6789/0}
> 2012-01-09 12:28:07.746431 mon e2: 2 mons at
> {ceph1=10.3.1.33:6789/0,ceph2=10.3.1.34:6789/0}
> 2012-01-09 12:28:07.746440 mon e3: 1 mons at {ceph1=10.3.1.33:6789/0}
> 2012-01-09 12:28:07.746446 mon e4: 2 mons at
> {ceph1=10.3.1.33:6789/0,ceph2=10.3.1.34:6789/0}
> 2012-01-09 12:28:07.746453 mon e5: 3 mons at
> {ceph1=10.3.1.33:6789/0,ceph2=10.3.1.34:6789/0,ceph3=10.3.1.35:6789/0}
>
> Guido
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Available space shrinks to zero after upgrading cluster
2012-01-09 23:40 ` Samuel Just
@ 2012-01-11 15:10 ` Guido Winkelmann
0 siblings, 0 replies; 4+ messages in thread
From: Guido Winkelmann @ 2012-01-11 15:10 UTC (permalink / raw)
To: Samuel Just; +Cc: ceph-devel
Am Montag, 9. Januar 2012, 15:40:10 schrieb Samuel Just:
> After restarting the OSDs, does the available space make sense?
> -Sam
Yes, it does, and the clients start working again, too. Still very odd though.
Guido
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2012-01-11 15:10 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-01-09 11:23 Available space shrinks to zero after upgrading cluster Guido Winkelmann
2012-01-09 11:28 ` Guido Winkelmann
2012-01-09 23:40 ` Samuel Just
2012-01-11 15:10 ` Guido Winkelmann
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.