* Re: accident corruption in osdmap
2012-08-09 15:53 ` Sage Weil
@ 2012-08-09 19:29 ` Stefan Kleijkers
2012-08-10 2:24 ` Eric_YH_Chen
1 sibling, 0 replies; 5+ messages in thread
From: Stefan Kleijkers @ 2012-08-09 19:29 UTC (permalink / raw)
To: Eric_YH_Chen; +Cc: Sage Weil, ceph-devel
Hello Eric,
We had the same problem and upgraded to the 3.5 kernel. The messages
disappeared from dmesg. We had a 3.4.x running. Don't know the exact
number anymore, but it was a low one (2 or something). So it seems that
upgrading the kernel fixes the problem.
Stefan
On 08/09/2012 05:53 PM, Sage Weil wrote:
> On Thu, 9 Aug 2012, Eric_YH_Chen@wiwynn.com wrote:
>> Dear all:
>>
>> My Environment: two servers, and 12 hard-disk on each server.
>> Version: Ceph 0.48, Kernel: 3.2.0-27
>>
>> We create a ceph cluster with 24 osd, 3 monitors
>> Osd.0 ~ osd.11 is on server1
>> Osd.12 ~ osd.23 is on server2
>> Mon.0 is on server1
>> Mon.1 is on server2
>> Mon.2 is on server3 which has no osd
>>
>> We create a rbd device and mount it as ext4 file system.
>> During read/write data on the rbd device, one of the storage server is shutdown by accident.
>> After reboot the server, we cannot access the rbd device any more.
>> One of the log shows the osdmap is corrupted.
>>
>> Aug 5 15:37:24 ubuntu-002 kernel: [78579.998582] libceph: corrupt inc osdmap epoch 78 off 98 (ffffc9000177d07e of ffffc9000177d01c-ffffc9000177edf2)
>>
>> We would like to know what kind of scenario would cause the corruption of osdmap and how to avoid it?
>> It seems that osdmap corruption cannot be recovered by the ceph cluster itself.
> Which kernel version are you running?
>
>> Is it the same issue with http://tracker.newdream.net/issues/2446?
>> In which version of kernel that we can find this patch? Thanks!
> If it is that issue, simply restarting the client is a workaround. It may
> crop up again on a newer epoch as the cluster recovers/rebalances data,
> but in each case a client restart will get past it.
>
> If that doesn't work, please attach a copy of the incremental osdmap
> ($mon_data/osdmap/78 from one of your monitors) and we can see what the
> actual corruption is.
>
> Thanks!
> sage
>
>>
>> ============= /var/log/kern.log =================================================================
>> Aug 5 15:31:44 ubuntu-002 kernel: [78240.712542] libceph: osd11 down
>> Aug 5 15:31:49 ubuntu-002 kernel: [78244.817151] libceph: osd12 down
>> Aug 5 15:31:52 ubuntu-002 kernel: [78248.151815] libceph: osd13 down
>> Aug 5 15:31:52 ubuntu-002 kernel: [78248.151913] libceph: osd14 down
>> Aug 5 15:31:53 ubuntu-002 kernel: [78249.250991] libceph: get_reply unknown tid 96452 from osd7
>> Aug 5 15:31:59 ubuntu-002 kernel: [78254.833033] libceph: osd15 down
>> Aug 5 15:31:59 ubuntu-002 kernel: [78254.833037] libceph: osd16 down
>> Aug 5 15:31:59 ubuntu-002 kernel: [78254.833039] libceph: osd17 down
>> Aug 5 15:31:59 ubuntu-002 kernel: [78254.833040] libceph: osd18 down
>> Aug 5 15:31:59 ubuntu-002 kernel: [78254.833042] libceph: osd19 down
>> Aug 5 15:31:59 ubuntu-002 kernel: [78254.833062] libceph: osd20 down
>> Aug 5 15:31:59 ubuntu-002 kernel: [78254.833064] libceph: osd21 down
>> Aug 5 15:36:46 ubuntu-002 kernel: [78541.813963] libceph: osd11 weight 0x0 (out)
>> Aug 5 15:37:09 ubuntu-002 kernel: [78564.811236] libceph: osd12 weight 0x0 (out)
>> Aug 5 15:37:09 ubuntu-002 kernel: [78564.811238] libceph: osd13 weight 0x0 (out)
>> Aug 5 15:37:09 ubuntu-002 kernel: [78564.811264] libceph: osd14 weight 0x0 (out)
>> Aug 5 15:37:09 ubuntu-002 kernel: [78564.811265] libceph: osd15 weight 0x0 (out)
>> Aug 5 15:37:09 ubuntu-002 kernel: [78564.811266] libceph: osd16 weight 0x0 (out)
>> Aug 5 15:37:09 ubuntu-002 kernel: [78564.811271] libceph: osd17 weight 0x0 (out)
>> Aug 5 15:37:09 ubuntu-002 kernel: [78564.811272] libceph: osd18 weight 0x0 (out)
>> Aug 5 15:37:09 ubuntu-002 kernel: [78564.811273] libceph: osd19 weight 0x0 (out)
>> Aug 5 15:37:09 ubuntu-002 kernel: [78564.811314] libceph: osd20 weight 0x0 (out)
>> Aug 5 15:37:09 ubuntu-002 kernel: [78564.811315] libceph: osd21 weight 0x0 (out)
>> Aug 5 15:37:24 ubuntu-002 kernel: [78579.998582] libceph: corrupt inc osdmap epoch 78 off 98 (ffffc9000177d07e of ffffc9000177d01c-ffffc9000177edf2)
>> Aug 5 15:37:24 ubuntu-002 kernel: [78579.998737] osdmap: 00000000: 05 00 70 d6 52 f9 b3 cc 44 c5 a2 eb c1 33 1d a2 ..p.R...D....3..
>> Aug 5 15:37:24 ubuntu-002 kernel: [78579.998739] osdmap: 00000010: 45 3d 4e 00 00 00 b3 22 1e 50 d0 b3 f3 2d ff ff E=N....".P...-..
>> Aug 5 15:37:24 ubuntu-002 kernel: [78579.998742] osdmap: 00000020: ff ff ff ff ff ff 00 00 00 00 00 00 00 00 ff ff ................
>> ...
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* RE: accident corruption in osdmap
2012-08-09 15:53 ` Sage Weil
2012-08-09 19:29 ` Stefan Kleijkers
@ 2012-08-10 2:24 ` Eric_YH_Chen
1 sibling, 0 replies; 5+ messages in thread
From: Eric_YH_Chen @ 2012-08-10 2:24 UTC (permalink / raw)
To: sage, dan.mick; +Cc: ceph-devel, Chris_YT_Huang
Dear Sage and Dan:
Thanks for kindly reply.
If we want to patch http://tracker.newdream.net/issues/2446 on kernel 3.2,
can we just modify the file as below?
We are afraid that it may cause other side effect.
diff --git a/net/ceph/osdmap.c b/net/ceph/osdmap.c
index 2592f3c..011a3a9 100644
--- a/net/ceph/osdmap.c
+++ b/net/ceph/osdmap.c
@@ -883,8 +883,12 @@ struct ceph_osdmap *osdmap_apply_incremental(void **p, void *end,
pglen = ceph_decode_32(p);
if (pglen) {
- /* insert */
ceph_decode_need(p, end, pglen*sizeof(u32), bad);
+
+ /* removing existing (if any) */
+ __remove_pg_mapping(&map->pg_temp, pgid);
+
+ /* insert */
pg = kmalloc(sizeof(*pg) + sizeof(u32)*pglen, GFP_NOFS);
if (!pg) {
err = -ENOMEM;
-----Original Message-----
From: Sage Weil [mailto:sage@inktank.com]
Sent: Thursday, August 09, 2012 11:54 PM
To: Eric YH Chen/WYHQ/Wiwynn
Cc: ceph-devel@vger.kernel.org
Subject: Re: accident corruption in osdmap
On Thu, 9 Aug 2012, Eric_YH_Chen@wiwynn.com wrote:
> Dear all:
>
> My Environment: two servers, and 12 hard-disk on each server.
> Version: Ceph 0.48, Kernel: 3.2.0-27
>
> We create a ceph cluster with 24 osd, 3 monitors
> Osd.0 ~ osd.11 is on server1
> Osd.12 ~ osd.23 is on server2
> Mon.0 is on server1
> Mon.1 is on server2
> Mon.2 is on server3 which has no osd
>
> We create a rbd device and mount it as ext4 file system.
> During read/write data on the rbd device, one of the storage server is shutdown by accident.
> After reboot the server, we cannot access the rbd device any more.
> One of the log shows the osdmap is corrupted.
>
> Aug 5 15:37:24 ubuntu-002 kernel: [78579.998582] libceph: corrupt inc
> osdmap epoch 78 off 98 (ffffc9000177d07e of
> ffffc9000177d01c-ffffc9000177edf2)
>
> We would like to know what kind of scenario would cause the corruption of osdmap and how to avoid it?
> It seems that osdmap corruption cannot be recovered by the ceph cluster itself.
Which kernel version are you running?
> Is it the same issue with http://tracker.newdream.net/issues/2446?
> In which version of kernel that we can find this patch? Thanks!
If it is that issue, simply restarting the client is a workaround. It may crop up again on a newer epoch as the cluster recovers/rebalances data, but in each case a client restart will get past it.
If that doesn't work, please attach a copy of the incremental osdmap
($mon_data/osdmap/78 from one of your monitors) and we can see what the actual corruption is.
Thanks!
sage
>
>
> ============= /var/log/kern.log
> =================================================================
> Aug 5 15:31:44 ubuntu-002 kernel: [78240.712542] libceph: osd11 down
> Aug 5 15:31:49 ubuntu-002 kernel: [78244.817151] libceph: osd12 down
> Aug 5 15:31:52 ubuntu-002 kernel: [78248.151815] libceph: osd13 down
> Aug 5 15:31:52 ubuntu-002 kernel: [78248.151913] libceph: osd14 down
> Aug 5 15:31:53 ubuntu-002 kernel: [78249.250991] libceph: get_reply
> unknown tid 96452 from osd7 Aug 5 15:31:59 ubuntu-002 kernel:
> [78254.833033] libceph: osd15 down Aug 5 15:31:59 ubuntu-002 kernel:
> [78254.833037] libceph: osd16 down Aug 5 15:31:59 ubuntu-002 kernel:
> [78254.833039] libceph: osd17 down Aug 5 15:31:59 ubuntu-002 kernel:
> [78254.833040] libceph: osd18 down Aug 5 15:31:59 ubuntu-002 kernel:
> [78254.833042] libceph: osd19 down Aug 5 15:31:59 ubuntu-002 kernel:
> [78254.833062] libceph: osd20 down Aug 5 15:31:59 ubuntu-002 kernel:
> [78254.833064] libceph: osd21 down Aug 5 15:36:46 ubuntu-002 kernel:
> [78541.813963] libceph: osd11 weight 0x0 (out) Aug 5 15:37:09
> ubuntu-002 kernel: [78564.811236] libceph: osd12 weight 0x0 (out) Aug
> 5 15:37:09 ubuntu-002 kernel: [78564.811238] libceph: osd13 weight 0x0
> (out) Aug 5 15:37:09 ubuntu-002 kernel: [78564.811264] libceph: osd14
> weight 0x0 (out) Aug 5 15:37:09 ubuntu-002 kernel: [78564.811265]
> libceph: osd15 weight 0x0 (out) Aug 5 15:37:09 ubuntu-002 kernel:
> [78564.811266] libceph: osd16 weight 0x0 (out) Aug 5 15:37:09
> ubuntu-002 kernel: [78564.811271] libceph: osd17 weight 0x0 (out) Aug
> 5 15:37:09 ubuntu-002 kernel: [78564.811272] libceph: osd18 weight 0x0
> (out) Aug 5 15:37:09 ubuntu-002 kernel: [78564.811273] libceph: osd19
> weight 0x0 (out) Aug 5 15:37:09 ubuntu-002 kernel: [78564.811314]
> libceph: osd20 weight 0x0 (out) Aug 5 15:37:09 ubuntu-002 kernel:
> [78564.811315] libceph: osd21 weight 0x0 (out) Aug 5 15:37:24
> ubuntu-002 kernel: [78579.998582] libceph: corrupt inc osdmap epoch 78
> off 98 (ffffc9000177d07e of ffffc9000177d01c-ffffc9000177edf2)
> Aug 5 15:37:24 ubuntu-002 kernel: [78579.998737] osdmap: 00000000: 05 00 70 d6 52 f9 b3 cc 44 c5 a2 eb c1 33 1d a2 ..p.R...D....3..
> Aug 5 15:37:24 ubuntu-002 kernel: [78579.998739] osdmap: 00000010: 45 3d 4e 00 00 00 b3 22 1e 50 d0 b3 f3 2d ff ff E=N....".P...-..
> Aug 5 15:37:24 ubuntu-002 kernel: [78579.998742] osdmap: 00000020: ff ff ff ff ff ff 00 00 00 00 00 00 00 00 ff ff ................
> ...
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org More majordomo
> info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply related [flat|nested] 5+ messages in thread