KVM/RBD Block device hangs

KVM/RBD Block device hangs

[Stefan Priebe - Profihost AG]

Hi List,

i'm testing the ceph / rbd block device with qemu and KVM. The creation
of a new disk went fine and i can also access the disk from the KVM client.

But after a short time while installing debian squeeze the osd seems to
crash and i see /core files on my machines.

What's next / wrong?

Greets
Stefan

Re: KVM/RBD Block device hangs

[Wido den Hollander]

Hi,

On 05/21/2012 04:44 PM, Stefan Priebe - Profihost AG wrote:
> Hi List,
>
> i'm testing the ceph / rbd block device with qemu and KVM. The creation
> of a new disk went fine and i can also access the disk from the KVM client.
>
> But after a short time while installing debian squeeze the osd seems to
> crash and i see /core files on my machines.

What do you mean the OSD crashes? I assume you have multiple OSD's in 
your cluster and one of them crashes? Or do ALL of them do go down?

>
> What's next / wrong?

Could you fetch the last 100 line from the log file and also tell us 
which version you are running?

Also, the output of "ceph -s" might be useful.

Wido

>
> Greets
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: KVM/RBD Block device hangs

[Stefan Priebe - Profihost AG]

Am 21.05.2012 16:47, schrieb Wido den Hollander:
> What do you mean the OSD crashes? I assume you have multiple OSD's in
> your cluster and one of them crashes? Or do ALL of them do go down?
2 of them went down (1st then 10 minutes later the 2nd)

>> What's next / wrong?
> 
> Could you fetch the last 100 line from the log file and also tell us
> which version you are running?
i'm upgrading right now to v0.47 and will try to reproduce.

> Also, the output of "ceph -s" might be useful.
before or after the crash?

Stefan

Re: KVM/RBD Block device hangs

[Wido den Hollander]

On 05/21/2012 04:50 PM, Stefan Priebe - Profihost AG wrote:
> Am 21.05.2012 16:47, schrieb Wido den Hollander:
>> What do you mean the OSD crashes? I assume you have multiple OSD's in
>> your cluster and one of them crashes? Or do ALL of them do go down?
> 2 of them went down (1st then 10 minutes later the 2nd)
>
>>> What's next / wrong?
>>
>> Could you fetch the last 100 line from the log file and also tell us
>> which version you are running?
> i'm upgrading right now to v0.47 and will try to reproduce.
>
>> Also, the output of "ceph -s" might be useful.
> before or after the crash?

Probably after, but both is fine. I just want to know how your cluster 
is doing and what the PG states are.

Wido

>
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: KVM/RBD Block device hangs

[Stefan Priebe]

Am 21.05.2012 16:59, schrieb Wido den Hollander:
> Probably after, but both is fine. I just want to know how your cluster
> is doing and what the PG states are.
will do so.

dmesg says at least:
[ 4975.850765] ceph-osd[6226]: segfault at 439d34000 ip 00007f0db39bfb23 
sp 00007f0da8c5e240 error 4 in libtcmalloc.so.0.0.0[7f0db399b000+3d000]

Stefan

Re: KVM/RBD Block device hangs

[Gregory Farnum]

On Mon, May 21, 2012 at 1:51 PM, Stefan Priebe <s.priebe@profihost.ag> wrote:
> Am 21.05.2012 16:59, schrieb Wido den Hollander:
>
>> Probably after, but both is fine. I just want to know how your cluster
>> is doing and what the PG states are.
>
> will do so.
>
> dmesg says at least:
> [ 4975.850765] ceph-osd[6226]: segfault at 439d34000 ip 00007f0db39bfb23 sp
> 00007f0da8c5e240 error 4 in libtcmalloc.so.0.0.0[7f0db399b000+3d000]

This is not an error we've seen before, and it's a little scary. Can you
1) run gdb and get a backtrace and a printout of each frame in the trace?
2) reproduce with high-level OSD debug logging enabled?

Re: KVM/RBD Block device hangs

[Stefan Priebe - Profihost AG]

Am 21.05.2012 22:57, schrieb Gregory Farnum:
> On Mon, May 21, 2012 at 1:51 PM, Stefan Priebe <s.priebe@profihost.ag> wrote:
>> Am 21.05.2012 16:59, schrieb Wido den Hollander:
>>
>>> Probably after, but both is fine. I just want to know how your cluster
>>> is doing and what the PG states are.
>>
>> will do so.
>>
>> dmesg says at least:
>> [ 4975.850765] ceph-osd[6226]: segfault at 439d34000 ip 00007f0db39bfb23 sp
>> 00007f0da8c5e240 error 4 in libtcmalloc.so.0.0.0[7f0db399b000+3d000]
> 
> This is not an error we've seen before, and it's a little scary. Can you
> 1) run gdb and get a backtrace and a printout of each frame in the trace?
> 2) reproduce with high-level OSD debug logging enabled?

I haven't seen this problem again with v0.47 i will report back if i see
it again.

Stefan