* accident corruption in osdmap
@ 2012-08-09 3:19 Eric_YH_Chen
2012-08-09 3:53 ` Dan Mick
2012-08-09 15:53 ` Sage Weil
0 siblings, 2 replies; 5+ messages in thread
From: Eric_YH_Chen @ 2012-08-09 3:19 UTC (permalink / raw)
To: ceph-devel
Dear all:
My Environment: two servers, and 12 hard-disk on each server.
Version: Ceph 0.48, Kernel: 3.2.0-27
We create a ceph cluster with 24 osd, 3 monitors
Osd.0 ~ osd.11 is on server1
Osd.12 ~ osd.23 is on server2
Mon.0 is on server1
Mon.1 is on server2
Mon.2 is on server3 which has no osd
We create a rbd device and mount it as ext4 file system.
During read/write data on the rbd device, one of the storage server is shutdown by accident.
After reboot the server, we cannot access the rbd device any more.
One of the log shows the osdmap is corrupted.
Aug 5 15:37:24 ubuntu-002 kernel: [78579.998582] libceph: corrupt inc osdmap epoch 78 off 98 (ffffc9000177d07e of ffffc9000177d01c-ffffc9000177edf2)
We would like to know what kind of scenario would cause the corruption of osdmap and how to avoid it?
It seems that osdmap corruption cannot be recovered by the ceph cluster itself.
Is it the same issue with http://tracker.newdream.net/issues/2446?
In which version of kernel that we can find this patch? Thanks!
============= /var/log/kern.log =================================================================
Aug 5 15:31:44 ubuntu-002 kernel: [78240.712542] libceph: osd11 down
Aug 5 15:31:49 ubuntu-002 kernel: [78244.817151] libceph: osd12 down
Aug 5 15:31:52 ubuntu-002 kernel: [78248.151815] libceph: osd13 down
Aug 5 15:31:52 ubuntu-002 kernel: [78248.151913] libceph: osd14 down
Aug 5 15:31:53 ubuntu-002 kernel: [78249.250991] libceph: get_reply unknown tid 96452 from osd7
Aug 5 15:31:59 ubuntu-002 kernel: [78254.833033] libceph: osd15 down
Aug 5 15:31:59 ubuntu-002 kernel: [78254.833037] libceph: osd16 down
Aug 5 15:31:59 ubuntu-002 kernel: [78254.833039] libceph: osd17 down
Aug 5 15:31:59 ubuntu-002 kernel: [78254.833040] libceph: osd18 down
Aug 5 15:31:59 ubuntu-002 kernel: [78254.833042] libceph: osd19 down
Aug 5 15:31:59 ubuntu-002 kernel: [78254.833062] libceph: osd20 down
Aug 5 15:31:59 ubuntu-002 kernel: [78254.833064] libceph: osd21 down
Aug 5 15:36:46 ubuntu-002 kernel: [78541.813963] libceph: osd11 weight 0x0 (out)
Aug 5 15:37:09 ubuntu-002 kernel: [78564.811236] libceph: osd12 weight 0x0 (out)
Aug 5 15:37:09 ubuntu-002 kernel: [78564.811238] libceph: osd13 weight 0x0 (out)
Aug 5 15:37:09 ubuntu-002 kernel: [78564.811264] libceph: osd14 weight 0x0 (out)
Aug 5 15:37:09 ubuntu-002 kernel: [78564.811265] libceph: osd15 weight 0x0 (out)
Aug 5 15:37:09 ubuntu-002 kernel: [78564.811266] libceph: osd16 weight 0x0 (out)
Aug 5 15:37:09 ubuntu-002 kernel: [78564.811271] libceph: osd17 weight 0x0 (out)
Aug 5 15:37:09 ubuntu-002 kernel: [78564.811272] libceph: osd18 weight 0x0 (out)
Aug 5 15:37:09 ubuntu-002 kernel: [78564.811273] libceph: osd19 weight 0x0 (out)
Aug 5 15:37:09 ubuntu-002 kernel: [78564.811314] libceph: osd20 weight 0x0 (out)
Aug 5 15:37:09 ubuntu-002 kernel: [78564.811315] libceph: osd21 weight 0x0 (out)
Aug 5 15:37:24 ubuntu-002 kernel: [78579.998582] libceph: corrupt inc osdmap epoch 78 off 98 (ffffc9000177d07e of ffffc9000177d01c-ffffc9000177edf2)
Aug 5 15:37:24 ubuntu-002 kernel: [78579.998737] osdmap: 00000000: 05 00 70 d6 52 f9 b3 cc 44 c5 a2 eb c1 33 1d a2 ..p.R...D....3..
Aug 5 15:37:24 ubuntu-002 kernel: [78579.998739] osdmap: 00000010: 45 3d 4e 00 00 00 b3 22 1e 50 d0 b3 f3 2d ff ff E=N....".P...-..
Aug 5 15:37:24 ubuntu-002 kernel: [78579.998742] osdmap: 00000020: ff ff ff ff ff ff 00 00 00 00 00 00 00 00 ff ff ................
...
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: accident corruption in osdmap
2012-08-09 3:19 accident corruption in osdmap Eric_YH_Chen
@ 2012-08-09 3:53 ` Dan Mick
2012-08-09 15:53 ` Sage Weil
1 sibling, 0 replies; 5+ messages in thread
From: Dan Mick @ 2012-08-09 3:53 UTC (permalink / raw)
To: Eric_YH_Chen; +Cc: ceph-devel
On 08/08/2012 08:19 PM, Eric_YH_Chen@wiwynn.com wrote:
> Dear all:
>
> My Environment: two servers, and 12 hard-disk on each server.
> Version: Ceph 0.48, Kernel: 3.2.0-27
>
> We create a ceph cluster with 24 osd, 3 monitors
> Osd.0 ~ osd.11 is on server1
> Osd.12 ~ osd.23 is on server2
> Mon.0 is on server1
> Mon.1 is on server2
> Mon.2 is on server3 which has no osd
>
> We create a rbd device and mount it as ext4 file system.
> During read/write data on the rbd device, one of the storage server is shutdown by accident.
> After reboot the server, we cannot access the rbd device any more.
> One of the log shows the osdmap is corrupted.
>
> Aug 5 15:37:24 ubuntu-002 kernel: [78579.998582] libceph: corrupt inc osdmap epoch 78 off 98 (ffffc9000177d07e of ffffc9000177d01c-ffffc9000177edf2)
>
> We would like to know what kind of scenario would cause the corruption of osdmap and how to avoid it?
> It seems that osdmap corruption cannot be recovered by the ceph cluster itself.
>
> Is it the same issue with http://tracker.newdream.net/issues/2446?
> In which version of kernel that we can find this patch? Thanks!
I'm not sure if it's the same issue, but looking at the Linux kernel
tree, it seems as though that fix is in v3.5-rc1
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: accident corruption in osdmap
2012-08-09 3:19 accident corruption in osdmap Eric_YH_Chen
2012-08-09 3:53 ` Dan Mick
@ 2012-08-09 15:53 ` Sage Weil
2012-08-09 19:29 ` Stefan Kleijkers
2012-08-10 2:24 ` Eric_YH_Chen
1 sibling, 2 replies; 5+ messages in thread
From: Sage Weil @ 2012-08-09 15:53 UTC (permalink / raw)
To: Eric_YH_Chen; +Cc: ceph-devel
On Thu, 9 Aug 2012, Eric_YH_Chen@wiwynn.com wrote:
> Dear all:
>
> My Environment: two servers, and 12 hard-disk on each server.
> Version: Ceph 0.48, Kernel: 3.2.0-27
>
> We create a ceph cluster with 24 osd, 3 monitors
> Osd.0 ~ osd.11 is on server1
> Osd.12 ~ osd.23 is on server2
> Mon.0 is on server1
> Mon.1 is on server2
> Mon.2 is on server3 which has no osd
>
> We create a rbd device and mount it as ext4 file system.
> During read/write data on the rbd device, one of the storage server is shutdown by accident.
> After reboot the server, we cannot access the rbd device any more.
> One of the log shows the osdmap is corrupted.
>
> Aug 5 15:37:24 ubuntu-002 kernel: [78579.998582] libceph: corrupt inc osdmap epoch 78 off 98 (ffffc9000177d07e of ffffc9000177d01c-ffffc9000177edf2)
>
> We would like to know what kind of scenario would cause the corruption of osdmap and how to avoid it?
> It seems that osdmap corruption cannot be recovered by the ceph cluster itself.
Which kernel version are you running?
> Is it the same issue with http://tracker.newdream.net/issues/2446?
> In which version of kernel that we can find this patch? Thanks!
If it is that issue, simply restarting the client is a workaround. It may
crop up again on a newer epoch as the cluster recovers/rebalances data,
but in each case a client restart will get past it.
If that doesn't work, please attach a copy of the incremental osdmap
($mon_data/osdmap/78 from one of your monitors) and we can see what the
actual corruption is.
Thanks!
sage
>
>
> ============= /var/log/kern.log =================================================================
> Aug 5 15:31:44 ubuntu-002 kernel: [78240.712542] libceph: osd11 down
> Aug 5 15:31:49 ubuntu-002 kernel: [78244.817151] libceph: osd12 down
> Aug 5 15:31:52 ubuntu-002 kernel: [78248.151815] libceph: osd13 down
> Aug 5 15:31:52 ubuntu-002 kernel: [78248.151913] libceph: osd14 down
> Aug 5 15:31:53 ubuntu-002 kernel: [78249.250991] libceph: get_reply unknown tid 96452 from osd7
> Aug 5 15:31:59 ubuntu-002 kernel: [78254.833033] libceph: osd15 down
> Aug 5 15:31:59 ubuntu-002 kernel: [78254.833037] libceph: osd16 down
> Aug 5 15:31:59 ubuntu-002 kernel: [78254.833039] libceph: osd17 down
> Aug 5 15:31:59 ubuntu-002 kernel: [78254.833040] libceph: osd18 down
> Aug 5 15:31:59 ubuntu-002 kernel: [78254.833042] libceph: osd19 down
> Aug 5 15:31:59 ubuntu-002 kernel: [78254.833062] libceph: osd20 down
> Aug 5 15:31:59 ubuntu-002 kernel: [78254.833064] libceph: osd21 down
> Aug 5 15:36:46 ubuntu-002 kernel: [78541.813963] libceph: osd11 weight 0x0 (out)
> Aug 5 15:37:09 ubuntu-002 kernel: [78564.811236] libceph: osd12 weight 0x0 (out)
> Aug 5 15:37:09 ubuntu-002 kernel: [78564.811238] libceph: osd13 weight 0x0 (out)
> Aug 5 15:37:09 ubuntu-002 kernel: [78564.811264] libceph: osd14 weight 0x0 (out)
> Aug 5 15:37:09 ubuntu-002 kernel: [78564.811265] libceph: osd15 weight 0x0 (out)
> Aug 5 15:37:09 ubuntu-002 kernel: [78564.811266] libceph: osd16 weight 0x0 (out)
> Aug 5 15:37:09 ubuntu-002 kernel: [78564.811271] libceph: osd17 weight 0x0 (out)
> Aug 5 15:37:09 ubuntu-002 kernel: [78564.811272] libceph: osd18 weight 0x0 (out)
> Aug 5 15:37:09 ubuntu-002 kernel: [78564.811273] libceph: osd19 weight 0x0 (out)
> Aug 5 15:37:09 ubuntu-002 kernel: [78564.811314] libceph: osd20 weight 0x0 (out)
> Aug 5 15:37:09 ubuntu-002 kernel: [78564.811315] libceph: osd21 weight 0x0 (out)
> Aug 5 15:37:24 ubuntu-002 kernel: [78579.998582] libceph: corrupt inc osdmap epoch 78 off 98 (ffffc9000177d07e of ffffc9000177d01c-ffffc9000177edf2)
> Aug 5 15:37:24 ubuntu-002 kernel: [78579.998737] osdmap: 00000000: 05 00 70 d6 52 f9 b3 cc 44 c5 a2 eb c1 33 1d a2 ..p.R...D....3..
> Aug 5 15:37:24 ubuntu-002 kernel: [78579.998739] osdmap: 00000010: 45 3d 4e 00 00 00 b3 22 1e 50 d0 b3 f3 2d ff ff E=N....".P...-..
> Aug 5 15:37:24 ubuntu-002 kernel: [78579.998742] osdmap: 00000020: ff ff ff ff ff ff 00 00 00 00 00 00 00 00 ff ff ................
> ...
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: accident corruption in osdmap
2012-08-09 15:53 ` Sage Weil
@ 2012-08-09 19:29 ` Stefan Kleijkers
2012-08-10 2:24 ` Eric_YH_Chen
1 sibling, 0 replies; 5+ messages in thread
From: Stefan Kleijkers @ 2012-08-09 19:29 UTC (permalink / raw)
To: Eric_YH_Chen; +Cc: Sage Weil, ceph-devel
Hello Eric,
We had the same problem and upgraded to the 3.5 kernel. The messages
disappeared from dmesg. We had a 3.4.x running. Don't know the exact
number anymore, but it was a low one (2 or something). So it seems that
upgrading the kernel fixes the problem.
Stefan
On 08/09/2012 05:53 PM, Sage Weil wrote:
> On Thu, 9 Aug 2012, Eric_YH_Chen@wiwynn.com wrote:
>> Dear all:
>>
>> My Environment: two servers, and 12 hard-disk on each server.
>> Version: Ceph 0.48, Kernel: 3.2.0-27
>>
>> We create a ceph cluster with 24 osd, 3 monitors
>> Osd.0 ~ osd.11 is on server1
>> Osd.12 ~ osd.23 is on server2
>> Mon.0 is on server1
>> Mon.1 is on server2
>> Mon.2 is on server3 which has no osd
>>
>> We create a rbd device and mount it as ext4 file system.
>> During read/write data on the rbd device, one of the storage server is shutdown by accident.
>> After reboot the server, we cannot access the rbd device any more.
>> One of the log shows the osdmap is corrupted.
>>
>> Aug 5 15:37:24 ubuntu-002 kernel: [78579.998582] libceph: corrupt inc osdmap epoch 78 off 98 (ffffc9000177d07e of ffffc9000177d01c-ffffc9000177edf2)
>>
>> We would like to know what kind of scenario would cause the corruption of osdmap and how to avoid it?
>> It seems that osdmap corruption cannot be recovered by the ceph cluster itself.
> Which kernel version are you running?
>
>> Is it the same issue with http://tracker.newdream.net/issues/2446?
>> In which version of kernel that we can find this patch? Thanks!
> If it is that issue, simply restarting the client is a workaround. It may
> crop up again on a newer epoch as the cluster recovers/rebalances data,
> but in each case a client restart will get past it.
>
> If that doesn't work, please attach a copy of the incremental osdmap
> ($mon_data/osdmap/78 from one of your monitors) and we can see what the
> actual corruption is.
>
> Thanks!
> sage
>
>>
>> ============= /var/log/kern.log =================================================================
>> Aug 5 15:31:44 ubuntu-002 kernel: [78240.712542] libceph: osd11 down
>> Aug 5 15:31:49 ubuntu-002 kernel: [78244.817151] libceph: osd12 down
>> Aug 5 15:31:52 ubuntu-002 kernel: [78248.151815] libceph: osd13 down
>> Aug 5 15:31:52 ubuntu-002 kernel: [78248.151913] libceph: osd14 down
>> Aug 5 15:31:53 ubuntu-002 kernel: [78249.250991] libceph: get_reply unknown tid 96452 from osd7
>> Aug 5 15:31:59 ubuntu-002 kernel: [78254.833033] libceph: osd15 down
>> Aug 5 15:31:59 ubuntu-002 kernel: [78254.833037] libceph: osd16 down
>> Aug 5 15:31:59 ubuntu-002 kernel: [78254.833039] libceph: osd17 down
>> Aug 5 15:31:59 ubuntu-002 kernel: [78254.833040] libceph: osd18 down
>> Aug 5 15:31:59 ubuntu-002 kernel: [78254.833042] libceph: osd19 down
>> Aug 5 15:31:59 ubuntu-002 kernel: [78254.833062] libceph: osd20 down
>> Aug 5 15:31:59 ubuntu-002 kernel: [78254.833064] libceph: osd21 down
>> Aug 5 15:36:46 ubuntu-002 kernel: [78541.813963] libceph: osd11 weight 0x0 (out)
>> Aug 5 15:37:09 ubuntu-002 kernel: [78564.811236] libceph: osd12 weight 0x0 (out)
>> Aug 5 15:37:09 ubuntu-002 kernel: [78564.811238] libceph: osd13 weight 0x0 (out)
>> Aug 5 15:37:09 ubuntu-002 kernel: [78564.811264] libceph: osd14 weight 0x0 (out)
>> Aug 5 15:37:09 ubuntu-002 kernel: [78564.811265] libceph: osd15 weight 0x0 (out)
>> Aug 5 15:37:09 ubuntu-002 kernel: [78564.811266] libceph: osd16 weight 0x0 (out)
>> Aug 5 15:37:09 ubuntu-002 kernel: [78564.811271] libceph: osd17 weight 0x0 (out)
>> Aug 5 15:37:09 ubuntu-002 kernel: [78564.811272] libceph: osd18 weight 0x0 (out)
>> Aug 5 15:37:09 ubuntu-002 kernel: [78564.811273] libceph: osd19 weight 0x0 (out)
>> Aug 5 15:37:09 ubuntu-002 kernel: [78564.811314] libceph: osd20 weight 0x0 (out)
>> Aug 5 15:37:09 ubuntu-002 kernel: [78564.811315] libceph: osd21 weight 0x0 (out)
>> Aug 5 15:37:24 ubuntu-002 kernel: [78579.998582] libceph: corrupt inc osdmap epoch 78 off 98 (ffffc9000177d07e of ffffc9000177d01c-ffffc9000177edf2)
>> Aug 5 15:37:24 ubuntu-002 kernel: [78579.998737] osdmap: 00000000: 05 00 70 d6 52 f9 b3 cc 44 c5 a2 eb c1 33 1d a2 ..p.R...D....3..
>> Aug 5 15:37:24 ubuntu-002 kernel: [78579.998739] osdmap: 00000010: 45 3d 4e 00 00 00 b3 22 1e 50 d0 b3 f3 2d ff ff E=N....".P...-..
>> Aug 5 15:37:24 ubuntu-002 kernel: [78579.998742] osdmap: 00000020: ff ff ff ff ff ff 00 00 00 00 00 00 00 00 ff ff ................
>> ...
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* RE: accident corruption in osdmap
2012-08-09 15:53 ` Sage Weil
2012-08-09 19:29 ` Stefan Kleijkers
@ 2012-08-10 2:24 ` Eric_YH_Chen
1 sibling, 0 replies; 5+ messages in thread
From: Eric_YH_Chen @ 2012-08-10 2:24 UTC (permalink / raw)
To: sage, dan.mick; +Cc: ceph-devel, Chris_YT_Huang
Dear Sage and Dan:
Thanks for kindly reply.
If we want to patch http://tracker.newdream.net/issues/2446 on kernel 3.2,
can we just modify the file as below?
We are afraid that it may cause other side effect.
diff --git a/net/ceph/osdmap.c b/net/ceph/osdmap.c
index 2592f3c..011a3a9 100644
--- a/net/ceph/osdmap.c
+++ b/net/ceph/osdmap.c
@@ -883,8 +883,12 @@ struct ceph_osdmap *osdmap_apply_incremental(void **p, void *end,
pglen = ceph_decode_32(p);
if (pglen) {
- /* insert */
ceph_decode_need(p, end, pglen*sizeof(u32), bad);
+
+ /* removing existing (if any) */
+ __remove_pg_mapping(&map->pg_temp, pgid);
+
+ /* insert */
pg = kmalloc(sizeof(*pg) + sizeof(u32)*pglen, GFP_NOFS);
if (!pg) {
err = -ENOMEM;
-----Original Message-----
From: Sage Weil [mailto:sage@inktank.com]
Sent: Thursday, August 09, 2012 11:54 PM
To: Eric YH Chen/WYHQ/Wiwynn
Cc: ceph-devel@vger.kernel.org
Subject: Re: accident corruption in osdmap
On Thu, 9 Aug 2012, Eric_YH_Chen@wiwynn.com wrote:
> Dear all:
>
> My Environment: two servers, and 12 hard-disk on each server.
> Version: Ceph 0.48, Kernel: 3.2.0-27
>
> We create a ceph cluster with 24 osd, 3 monitors
> Osd.0 ~ osd.11 is on server1
> Osd.12 ~ osd.23 is on server2
> Mon.0 is on server1
> Mon.1 is on server2
> Mon.2 is on server3 which has no osd
>
> We create a rbd device and mount it as ext4 file system.
> During read/write data on the rbd device, one of the storage server is shutdown by accident.
> After reboot the server, we cannot access the rbd device any more.
> One of the log shows the osdmap is corrupted.
>
> Aug 5 15:37:24 ubuntu-002 kernel: [78579.998582] libceph: corrupt inc
> osdmap epoch 78 off 98 (ffffc9000177d07e of
> ffffc9000177d01c-ffffc9000177edf2)
>
> We would like to know what kind of scenario would cause the corruption of osdmap and how to avoid it?
> It seems that osdmap corruption cannot be recovered by the ceph cluster itself.
Which kernel version are you running?
> Is it the same issue with http://tracker.newdream.net/issues/2446?
> In which version of kernel that we can find this patch? Thanks!
If it is that issue, simply restarting the client is a workaround. It may crop up again on a newer epoch as the cluster recovers/rebalances data, but in each case a client restart will get past it.
If that doesn't work, please attach a copy of the incremental osdmap
($mon_data/osdmap/78 from one of your monitors) and we can see what the actual corruption is.
Thanks!
sage
>
>
> ============= /var/log/kern.log
> =================================================================
> Aug 5 15:31:44 ubuntu-002 kernel: [78240.712542] libceph: osd11 down
> Aug 5 15:31:49 ubuntu-002 kernel: [78244.817151] libceph: osd12 down
> Aug 5 15:31:52 ubuntu-002 kernel: [78248.151815] libceph: osd13 down
> Aug 5 15:31:52 ubuntu-002 kernel: [78248.151913] libceph: osd14 down
> Aug 5 15:31:53 ubuntu-002 kernel: [78249.250991] libceph: get_reply
> unknown tid 96452 from osd7 Aug 5 15:31:59 ubuntu-002 kernel:
> [78254.833033] libceph: osd15 down Aug 5 15:31:59 ubuntu-002 kernel:
> [78254.833037] libceph: osd16 down Aug 5 15:31:59 ubuntu-002 kernel:
> [78254.833039] libceph: osd17 down Aug 5 15:31:59 ubuntu-002 kernel:
> [78254.833040] libceph: osd18 down Aug 5 15:31:59 ubuntu-002 kernel:
> [78254.833042] libceph: osd19 down Aug 5 15:31:59 ubuntu-002 kernel:
> [78254.833062] libceph: osd20 down Aug 5 15:31:59 ubuntu-002 kernel:
> [78254.833064] libceph: osd21 down Aug 5 15:36:46 ubuntu-002 kernel:
> [78541.813963] libceph: osd11 weight 0x0 (out) Aug 5 15:37:09
> ubuntu-002 kernel: [78564.811236] libceph: osd12 weight 0x0 (out) Aug
> 5 15:37:09 ubuntu-002 kernel: [78564.811238] libceph: osd13 weight 0x0
> (out) Aug 5 15:37:09 ubuntu-002 kernel: [78564.811264] libceph: osd14
> weight 0x0 (out) Aug 5 15:37:09 ubuntu-002 kernel: [78564.811265]
> libceph: osd15 weight 0x0 (out) Aug 5 15:37:09 ubuntu-002 kernel:
> [78564.811266] libceph: osd16 weight 0x0 (out) Aug 5 15:37:09
> ubuntu-002 kernel: [78564.811271] libceph: osd17 weight 0x0 (out) Aug
> 5 15:37:09 ubuntu-002 kernel: [78564.811272] libceph: osd18 weight 0x0
> (out) Aug 5 15:37:09 ubuntu-002 kernel: [78564.811273] libceph: osd19
> weight 0x0 (out) Aug 5 15:37:09 ubuntu-002 kernel: [78564.811314]
> libceph: osd20 weight 0x0 (out) Aug 5 15:37:09 ubuntu-002 kernel:
> [78564.811315] libceph: osd21 weight 0x0 (out) Aug 5 15:37:24
> ubuntu-002 kernel: [78579.998582] libceph: corrupt inc osdmap epoch 78
> off 98 (ffffc9000177d07e of ffffc9000177d01c-ffffc9000177edf2)
> Aug 5 15:37:24 ubuntu-002 kernel: [78579.998737] osdmap: 00000000: 05 00 70 d6 52 f9 b3 cc 44 c5 a2 eb c1 33 1d a2 ..p.R...D....3..
> Aug 5 15:37:24 ubuntu-002 kernel: [78579.998739] osdmap: 00000010: 45 3d 4e 00 00 00 b3 22 1e 50 d0 b3 f3 2d ff ff E=N....".P...-..
> Aug 5 15:37:24 ubuntu-002 kernel: [78579.998742] osdmap: 00000020: ff ff ff ff ff ff 00 00 00 00 00 00 00 00 ff ff ................
> ...
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org More majordomo
> info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply related [flat|nested] 5+ messages in thread
end of thread, other threads:[~2012-08-10 2:24 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-08-09 3:19 accident corruption in osdmap Eric_YH_Chen
2012-08-09 3:53 ` Dan Mick
2012-08-09 15:53 ` Sage Weil
2012-08-09 19:29 ` Stefan Kleijkers
2012-08-10 2:24 ` Eric_YH_Chen
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.