qemu librbd patches

qemu librbd patches

[Christian Brunner]

Hi,

after upgrading to qemu with josh's librbd patches I'm triggering the
following assertion, when I'm running "yum upgrade" inside a guest.

I suspect there is a race condition somewhere, but I don't have a clue
at the moment.

Regards,
Christian

common/Mutex.h: In function 'void Mutex::Lock(bool)', in thread '0x7fea16b36700'
common/Mutex.h: 118: FAILED assert(r == 0)
common/Mutex.h: In function 'void Mutex::Lock(bool)', in thread '0x7fea16b36700'
common/Mutex.h: 118: FAILED assert(r == 0)
 ceph version 0.27.1 (commit:44900d4c13f02913b3347cac3e1dc33632d5b8ff)
 1: (librbd::aio_write(librbd::ImageCtx*, unsigned long, unsigned
long, char const*, librbd::AioCompletion*)+0x54e) [0x7fea2299523e]
 2: /usr/bin/qemu-kvm() [0x46a165]
 3: /usr/bin/qemu-kvm() [0x46a1d0]
 4: /usr/bin/qemu-kvm() [0x44349f]
 ceph version 0.27.1 (commit:44900d4c13f02913b3347cac3e1dc33632d5b8ff)
 1: (librbd::aio_write(librbd::ImageCtx*, unsigned long, unsigned
long, char const*, librbd::AioCompletion*)+0x54e) [0x7fea2299523e]
 2: /usr/bin/qemu-kvm() [0x46a165]
 3: /usr/bin/qemu-kvm() [0x46a1d0]
 4: /usr/bin/qemu-kvm() [0x44349f]
 ceph version 0.27.1 (commit:44900d4c13f02913b3347cac3e1dc33632d5b8ff)
 1: (librbd::aio_write(librbd::ImageCtx*, unsigned long, unsigned
long, char const*, librbd::AioCompletion*)+0x54e) [0x7fea2299523e]
 2: /usr/bin/qemu-kvm() [0x46a165]
 3: /usr/bin/qemu-kvm() [0x46a1d0]
 4: /usr/bin/qemu-kvm() [0x44349f]
 ceph version 0.27.1 (commit:44900d4c13f02913b3347cac3e1dc33632d5b8ff)
 1: (librbd::aio_write(librbd::ImageCtx*, unsigned long, unsigned
long, char const*, librbd::AioCompletion*)+0x54e) [0x7fea2299523e]
 2: /usr/bin/qemu-kvm() [0x46a165]
 3: /usr/bin/qemu-kvm() [0x46a1d0]
 4: /usr/bin/qemu-kvm() [0x44349f]
terminate called after throwing an instance of 'ceph::FailedAssertion'
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: qemu librbd patches

[Josh Durgin]

On Mon, 9 May 2011 10:59:41 +0200, Christian Brunner <chb@muc.de>
wrote:
> Hi,
> 
> after upgrading to qemu with josh's librbd patches I'm triggering the
> following assertion, when I'm running "yum upgrade" inside a guest.
> 
> I suspect there is a race condition somewhere, but I don't have a clue
> at the moment.

This is similar to a race condition Stefan encountered
(http://marc.info/?l=kvm&m=130225223223016&w=2),
although this crash was during a write whereas his test was just
reading the image.

> 
> Regards,
> Christian
> 
> common/Mutex.h: In function 'void Mutex::Lock(bool)', in thread
> '0x7fea16b36700'
> common/Mutex.h: 118: FAILED assert(r == 0)

Usually this assert fails due to a use-after-free. If you've got
a core file, could you print the lock that hit this assert and
the data structure containing it?

Thanks!
Josh

Re: qemu librbd patches

[Yehuda Sadeh Weinraub]

On Mon, May 9, 2011 at 8:22 AM, Josh Durgin <josh.durgin@dreamhost.com> wrote:
> On Mon, 9 May 2011 10:59:41 +0200, Christian Brunner <chb@muc.de>
> wrote:
>> Hi,
>>
>> after upgrading to qemu with josh's librbd patches I'm triggering the
>> following assertion, when I'm running "yum upgrade" inside a guest.
>>
>> I suspect there is a race condition somewhere, but I don't have a clue
>> at the moment.
>
> This is similar to a race condition Stefan encountered
> (http://marc.info/?l=kvm&m=130225223223016&w=2),
> although this crash was during a write whereas his test was just
> reading the image.
>
>>
>> Regards,
>> Christian
>>
>> common/Mutex.h: In function 'void Mutex::Lock(bool)', in thread
>> '0x7fea16b36700'
>> common/Mutex.h: 118: FAILED assert(r == 0)
>
> Usually this assert fails due to a use-after-free. If you've got
> a core file, could you print the lock that hit this assert and
> the data structure containing it?

Btw, probably not related but should be pointed out. There are a few
places in librbd where it fails to take the image context lock, and
it's probably racy. It's probably a big problem when resizing image,
but there might be other issues.

Yehuda

Re: qemu librbd patches

[Josh Durgin]

On 05/09/2011 01:59 AM, Christian Brunner wrote:
> Hi,
>
> after upgrading to qemu with josh's librbd patches I'm triggering the
> following assertion, when I'm running "yum upgrade" inside a guest.
>
> I suspect there is a race condition somewhere, but I don't have a clue
> at the moment.
>
> Regards,
> Christian
>
> common/Mutex.h: In function 'void Mutex::Lock(bool)', in thread '0x7fea16b36700'
> common/Mutex.h: 118: FAILED assert(r == 0)
> common/Mutex.h: In function 'void Mutex::Lock(bool)', in thread '0x7fea16b36700'
> common/Mutex.h: 118: FAILED assert(r == 0)
>   ceph version 0.27.1 (commit:44900d4c13f02913b3347cac3e1dc33632d5b8ff)
>   1: (librbd::aio_write(librbd::ImageCtx*, unsigned long, unsigned
> long, char const*, librbd::AioCompletion*)+0x54e) [0x7fea2299523e]
>   2: /usr/bin/qemu-kvm() [0x46a165]
>   3: /usr/bin/qemu-kvm() [0x46a1d0]
>   4: /usr/bin/qemu-kvm() [0x44349f]
>   ceph version 0.27.1 (commit:44900d4c13f02913b3347cac3e1dc33632d5b8ff)
>   1: (librbd::aio_write(librbd::ImageCtx*, unsigned long, unsigned
> long, char const*, librbd::AioCompletion*)+0x54e) [0x7fea2299523e]
>   2: /usr/bin/qemu-kvm() [0x46a165]
>   3: /usr/bin/qemu-kvm() [0x46a1d0]
>   4: /usr/bin/qemu-kvm() [0x44349f]
>   ceph version 0.27.1 (commit:44900d4c13f02913b3347cac3e1dc33632d5b8ff)
>   1: (librbd::aio_write(librbd::ImageCtx*, unsigned long, unsigned
> long, char const*, librbd::AioCompletion*)+0x54e) [0x7fea2299523e]
>   2: /usr/bin/qemu-kvm() [0x46a165]
>   3: /usr/bin/qemu-kvm() [0x46a1d0]
>   4: /usr/bin/qemu-kvm() [0x44349f]
>   ceph version 0.27.1 (commit:44900d4c13f02913b3347cac3e1dc33632d5b8ff)
>   1: (librbd::aio_write(librbd::ImageCtx*, unsigned long, unsigned
> long, char const*, librbd::AioCompletion*)+0x54e) [0x7fea2299523e]
>   2: /usr/bin/qemu-kvm() [0x46a165]
>   3: /usr/bin/qemu-kvm() [0x46a1d0]
>   4: /usr/bin/qemu-kvm() [0x44349f]
> terminate called after throwing an instance of 'ceph::FailedAssertion'

Hi Christian,

Could you try to get a core file for this? I still haven't been able to 
reproduce it.

Thanks,
Josh

Re: qemu librbd patches

[Christian Brunner]

I'm sorry. At the moment I don't have a core file available and my
colleagues don't allow me to reproduce it. ;-)

We should get a new server in one or two weeks. I will try to get a core then.

Christian

2011/5/24 Josh Durgin <josh.durgin@dreamhost.com>:
> On 05/09/2011 01:59 AM, Christian Brunner wrote:
>>
>> Hi,
>>
>> after upgrading to qemu with josh's librbd patches I'm triggering the
>> following assertion, when I'm running "yum upgrade" inside a guest.
>>
>> I suspect there is a race condition somewhere, but I don't have a clue
>> at the moment.
>>
>> Regards,
>> Christian
>>
>> common/Mutex.h: In function 'void Mutex::Lock(bool)', in thread
>> '0x7fea16b36700'
>> common/Mutex.h: 118: FAILED assert(r == 0)
>> common/Mutex.h: In function 'void Mutex::Lock(bool)', in thread
>> '0x7fea16b36700'
>> common/Mutex.h: 118: FAILED assert(r == 0)
>>  ceph version 0.27.1 (commit:44900d4c13f02913b3347cac3e1dc33632d5b8ff)
>>  1: (librbd::aio_write(librbd::ImageCtx*, unsigned long, unsigned
>> long, char const*, librbd::AioCompletion*)+0x54e) [0x7fea2299523e]
>>  2: /usr/bin/qemu-kvm() [0x46a165]
>>  3: /usr/bin/qemu-kvm() [0x46a1d0]
>>  4: /usr/bin/qemu-kvm() [0x44349f]
>>  ceph version 0.27.1 (commit:44900d4c13f02913b3347cac3e1dc33632d5b8ff)
>>  1: (librbd::aio_write(librbd::ImageCtx*, unsigned long, unsigned
>> long, char const*, librbd::AioCompletion*)+0x54e) [0x7fea2299523e]
>>  2: /usr/bin/qemu-kvm() [0x46a165]
>>  3: /usr/bin/qemu-kvm() [0x46a1d0]
>>  4: /usr/bin/qemu-kvm() [0x44349f]
>>  ceph version 0.27.1 (commit:44900d4c13f02913b3347cac3e1dc33632d5b8ff)
>>  1: (librbd::aio_write(librbd::ImageCtx*, unsigned long, unsigned
>> long, char const*, librbd::AioCompletion*)+0x54e) [0x7fea2299523e]
>>  2: /usr/bin/qemu-kvm() [0x46a165]
>>  3: /usr/bin/qemu-kvm() [0x46a1d0]
>>  4: /usr/bin/qemu-kvm() [0x44349f]
>>  ceph version 0.27.1 (commit:44900d4c13f02913b3347cac3e1dc33632d5b8ff)
>>  1: (librbd::aio_write(librbd::ImageCtx*, unsigned long, unsigned
>> long, char const*, librbd::AioCompletion*)+0x54e) [0x7fea2299523e]
>>  2: /usr/bin/qemu-kvm() [0x46a165]
>>  3: /usr/bin/qemu-kvm() [0x46a1d0]
>>  4: /usr/bin/qemu-kvm() [0x44349f]
>> terminate called after throwing an instance of 'ceph::FailedAssertion'
>
> Hi Christian,
>
> Could you try to get a core file for this? I still haven't been able to
> reproduce it.
>
> Thanks,
> Josh
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html