BUG: kvm crashing in void librbd::AioCompletion::complete_request

BUG: kvm crashing in void librbd::AioCompletion::complete_request

[Stefan Priebe]

Hello,

while benchmarking the new v2 rbd format i saw several crashes of KVM. 
First i thought this was kernel related but i was just happening faster 
with one kernel than the other.

The KVM crash / exit looks like this:
librbd/AioCompletion.cc: In function 'void 
librbd::AioCompletion::complete_request(CephContext*, ssize_t)' thread 
7f73db67c780 time 2012-11-04 14:59:44.997009
librbd/AioCompletion.cc: 43: FAILED assert(bl.length() == read_buf_len)
  ceph version 0.53-585-ga01b112 (a01b112d71a0b6a1bb206d53867d13536d17bbf6)
  1: (librbd::AioCompletion::complete_request(CephContext*, long)+0xa89) 
[0x7f73d9337319]
  2: (librbd::C_AioRead::finish(int)+0x174) [0x7f73d93374e4]
  3: (Context::complete(int)+0xa) [0x7f73d933796a]
  4: (librbd::C_CacheRead::finish(int)+0xe) [0x7f73d933683e]
  5: (Context::complete(int)+0xa) [0x7f73d933796a]
  6: (librbd::ImageCtx::aio_read_from_cache(object_t, 
ceph::buffer::list*, unsigned long, unsigned long, Context*)+0x3a0) 
[0x7f73d9347bd0]
  7: (librbd::aio_read(librbd::ImageCtx*, std::vector<std::pair<unsigned 
long, unsigned long>, std::allocator<std::pair<unsigned long, unsigned 
long> > > const&, char*, ceph::buffer::list*, 
librbd::AioCompletion*)+0x765) [0x7f73d9354995]
  8: (librbd::aio_read(librbd::ImageCtx*, unsigned long, unsigned long, 
char*, ceph::buffer::list*, librbd::AioCompletion*)+0xa2) [0x7f73d9354f92]
  9: (()+0x9f2de) [0x7f73db73d2de]
  10: (()+0x9f490) [0x7f73db73d490]
  11: (()+0x83c45) [0x7f73db721c45]
  12: (()+0x850ea) [0x7f73db7230ea]
  NOTE: a copy of the executable, or `objdump -rdS <executable>` is 
needed to interpret this.
terminate called after throwing an instance of 'ceph::FailedAssertion'

Stefan

Re: BUG: kvm crashing in void librbd::AioCompletion::complete_request

[Sage Weil]

On Sun, 4 Nov 2012, Stefan Priebe wrote:
> Hello,
> 
> while benchmarking the new v2 rbd format i saw several crashes of KVM. First i
> thought this was kernel related but i was just happening faster with one
> kernel than the other.

Can you try the wip-rbd-read branch?  It has a bunch of fixes that aren't 
in master yet.  I haven't seen this specific crash (I don't think), but I 
was reliable hitting other issues with master that wip-rbd-read reslves.

Alternatively, you could cherry-pick 
b1d866efbdf7e8a7bc083b1f3ea56f24e10c8fcd, which I'm guessing is your 
problem.

Thanks!
sage


> 
> The KVM crash / exit looks like this:
> librbd/AioCompletion.cc: In function 'void
> librbd::AioCompletion::complete_request(CephContext*, ssize_t)' thread
> 7f73db67c780 time 2012-11-04 14:59:44.997009
> librbd/AioCompletion.cc: 43: FAILED assert(bl.length() == read_buf_len)
>  ceph version 0.53-585-ga01b112 (a01b112d71a0b6a1bb206d53867d13536d17bbf6)
>  1: (librbd::AioCompletion::complete_request(CephContext*, long)+0xa89)
> [0x7f73d9337319]
>  2: (librbd::C_AioRead::finish(int)+0x174) [0x7f73d93374e4]
>  3: (Context::complete(int)+0xa) [0x7f73d933796a]
>  4: (librbd::C_CacheRead::finish(int)+0xe) [0x7f73d933683e]
>  5: (Context::complete(int)+0xa) [0x7f73d933796a]
>  6: (librbd::ImageCtx::aio_read_from_cache(object_t, ceph::buffer::list*,
> unsigned long, unsigned long, Context*)+0x3a0) [0x7f73d9347bd0]
>  7: (librbd::aio_read(librbd::ImageCtx*, std::vector<std::pair<unsigned long,
> unsigned long>, std::allocator<std::pair<unsigned long, unsigned long> > >
> const&, char*, ceph::buffer::list*, librbd::AioCompletion*)+0x765)
> [0x7f73d9354995]
>  8: (librbd::aio_read(librbd::ImageCtx*, unsigned long, unsigned long, char*,
> ceph::buffer::list*, librbd::AioCompletion*)+0xa2) [0x7f73d9354f92]
>  9: (()+0x9f2de) [0x7f73db73d2de]
>  10: (()+0x9f490) [0x7f73db73d490]
>  11: (()+0x83c45) [0x7f73db721c45]
>  12: (()+0x850ea) [0x7f73db7230ea]
>  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
> interpret this.
> terminate called after throwing an instance of 'ceph::FailedAssertion'
> 
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: BUG: kvm crashing in void librbd::AioCompletion::complete_request

[Stefan Priebe]

Can i merge wip-rbd-read into master?

Stefan
Am 04.11.2012 15:06, schrieb Sage Weil:
> On Sun, 4 Nov 2012, Stefan Priebe wrote:
>> Hello,
>>
>> while benchmarking the new v2 rbd format i saw several crashes of KVM. First i
>> thought this was kernel related but i was just happening faster with one
>> kernel than the other.
>
> Can you try the wip-rbd-read branch?  It has a bunch of fixes that aren't
> in master yet.  I haven't seen this specific crash (I don't think), but I
> was reliable hitting other issues with master that wip-rbd-read reslves.
>
> Alternatively, you could cherry-pick
> b1d866efbdf7e8a7bc083b1f3ea56f24e10c8fcd, which I'm guessing is your
> problem.
>
> Thanks!
> sage
>
>
>>
>> The KVM crash / exit looks like this:
>> librbd/AioCompletion.cc: In function 'void
>> librbd::AioCompletion::complete_request(CephContext*, ssize_t)' thread
>> 7f73db67c780 time 2012-11-04 14:59:44.997009
>> librbd/AioCompletion.cc: 43: FAILED assert(bl.length() == read_buf_len)
>>   ceph version 0.53-585-ga01b112 (a01b112d71a0b6a1bb206d53867d13536d17bbf6)
>>   1: (librbd::AioCompletion::complete_request(CephContext*, long)+0xa89)
>> [0x7f73d9337319]
>>   2: (librbd::C_AioRead::finish(int)+0x174) [0x7f73d93374e4]
>>   3: (Context::complete(int)+0xa) [0x7f73d933796a]
>>   4: (librbd::C_CacheRead::finish(int)+0xe) [0x7f73d933683e]
>>   5: (Context::complete(int)+0xa) [0x7f73d933796a]
>>   6: (librbd::ImageCtx::aio_read_from_cache(object_t, ceph::buffer::list*,
>> unsigned long, unsigned long, Context*)+0x3a0) [0x7f73d9347bd0]
>>   7: (librbd::aio_read(librbd::ImageCtx*, std::vector<std::pair<unsigned long,
>> unsigned long>, std::allocator<std::pair<unsigned long, unsigned long> > >
>> const&, char*, ceph::buffer::list*, librbd::AioCompletion*)+0x765)
>> [0x7f73d9354995]
>>   8: (librbd::aio_read(librbd::ImageCtx*, unsigned long, unsigned long, char*,
>> ceph::buffer::list*, librbd::AioCompletion*)+0xa2) [0x7f73d9354f92]
>>   9: (()+0x9f2de) [0x7f73db73d2de]
>>   10: (()+0x9f490) [0x7f73db73d490]
>>   11: (()+0x83c45) [0x7f73db721c45]
>>   12: (()+0x850ea) [0x7f73db7230ea]
>>   NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
>> interpret this.
>> terminate called after throwing an instance of 'ceph::FailedAssertion'
>>
>> Stefan
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>

Re: BUG: kvm crashing in void librbd::AioCompletion::complete_request

[Sage Weil]

On Sun, 4 Nov 2012, Stefan Priebe wrote:
> Can i merge wip-rbd-read into master?

Yeah.  I'm going to do a bit more testing first before I do it, but it 
should apply cleanly.  Hopefully later today.

sage

> 
> Stefan
> Am 04.11.2012 15:06, schrieb Sage Weil:
> > On Sun, 4 Nov 2012, Stefan Priebe wrote:
> > > Hello,
> > > 
> > > while benchmarking the new v2 rbd format i saw several crashes of KVM.
> > > First i
> > > thought this was kernel related but i was just happening faster with one
> > > kernel than the other.
> > 
> > Can you try the wip-rbd-read branch?  It has a bunch of fixes that aren't
> > in master yet.  I haven't seen this specific crash (I don't think), but I
> > was reliable hitting other issues with master that wip-rbd-read reslves.
> > 
> > Alternatively, you could cherry-pick
> > b1d866efbdf7e8a7bc083b1f3ea56f24e10c8fcd, which I'm guessing is your
> > problem.
> > 
> > Thanks!
> > sage
> > 
> > 
> > > 
> > > The KVM crash / exit looks like this:
> > > librbd/AioCompletion.cc: In function 'void
> > > librbd::AioCompletion::complete_request(CephContext*, ssize_t)' thread
> > > 7f73db67c780 time 2012-11-04 14:59:44.997009
> > > librbd/AioCompletion.cc: 43: FAILED assert(bl.length() == read_buf_len)
> > >   ceph version 0.53-585-ga01b112
> > > (a01b112d71a0b6a1bb206d53867d13536d17bbf6)
> > >   1: (librbd::AioCompletion::complete_request(CephContext*, long)+0xa89)
> > > [0x7f73d9337319]
> > >   2: (librbd::C_AioRead::finish(int)+0x174) [0x7f73d93374e4]
> > >   3: (Context::complete(int)+0xa) [0x7f73d933796a]
> > >   4: (librbd::C_CacheRead::finish(int)+0xe) [0x7f73d933683e]
> > >   5: (Context::complete(int)+0xa) [0x7f73d933796a]
> > >   6: (librbd::ImageCtx::aio_read_from_cache(object_t, ceph::buffer::list*,
> > > unsigned long, unsigned long, Context*)+0x3a0) [0x7f73d9347bd0]
> > >   7: (librbd::aio_read(librbd::ImageCtx*, std::vector<std::pair<unsigned
> > > long,
> > > unsigned long>, std::allocator<std::pair<unsigned long, unsigned long> > >
> > > const&, char*, ceph::buffer::list*, librbd::AioCompletion*)+0x765)
> > > [0x7f73d9354995]
> > >   8: (librbd::aio_read(librbd::ImageCtx*, unsigned long, unsigned long,
> > > char*,
> > > ceph::buffer::list*, librbd::AioCompletion*)+0xa2) [0x7f73d9354f92]
> > >   9: (()+0x9f2de) [0x7f73db73d2de]
> > >   10: (()+0x9f490) [0x7f73db73d490]
> > >   11: (()+0x83c45) [0x7f73db721c45]
> > >   12: (()+0x850ea) [0x7f73db7230ea]
> > >   NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed
> > > to
> > > interpret this.
> > > terminate called after throwing an instance of 'ceph::FailedAssertion'
> > > 
> > > Stefan
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > 
> > > 
> 
> 

Re: BUG: kvm crashing in void librbd::AioCompletion::complete_request

[Stefan Priebe]

Am 04.11.2012 15:12, schrieb Sage Weil:
> On Sun, 4 Nov 2012, Stefan Priebe wrote:
>> Can i merge wip-rbd-read into master?
>
> Yeah.  I'm going to do a bit more testing first before I do it, but it
> should apply cleanly.  Hopefully later today.

Any other branches to merge locally while benchmarking / testing?

Greets,
Stefan

Re: BUG: kvm crashing in void librbd::AioCompletion::complete_request

[Sage Weil]

On Sun, 4 Nov 2012, Stefan Priebe wrote:
> Am 04.11.2012 15:12, schrieb Sage Weil:
> > On Sun, 4 Nov 2012, Stefan Priebe wrote:
> > > Can i merge wip-rbd-read into master?
> > 
> > Yeah.  I'm going to do a bit more testing first before I do it, but it
> > should apply cleanly.  Hopefully later today.
> 
> Any other branches to merge locally while benchmarking / testing?

Nope, although you get the latest wip-rbd-read (I fixed an issue with 
glibc's random_r sucking).

sage

Re: BUG: kvm crashing in void librbd::AioCompletion::complete_request

[Stefan Priebe]

Am 04.11.2012 20:19, schrieb Sage Weil:
> On Sun, 4 Nov 2012, Stefan Priebe wrote:
>> Am 04.11.2012 15:12, schrieb Sage Weil:
>>> On Sun, 4 Nov 2012, Stefan Priebe wrote:
>>>> Can i merge wip-rbd-read into master?
>>>
>>> Yeah.  I'm going to do a bit more testing first before I do it, but it
>>> should apply cleanly.  Hopefully later today.
>>
>> Any other branches to merge locally while benchmarking / testing?
>
> Nope, although you get the latest wip-rbd-read (I fixed an issue with
> glibc's random_r sucking).

Did i get you wrong? There's no commit from sage today?
https://github.com/ceph/ceph/commits/wip-rbd-read

Stefan

Re: BUG: kvm crashing in void librbd::AioCompletion::complete_request

[Sage Weil]

On Sun, 4 Nov 2012, Stefan Priebe wrote:
> Am 04.11.2012 20:19, schrieb Sage Weil:
> > On Sun, 4 Nov 2012, Stefan Priebe wrote:
> > > Am 04.11.2012 15:12, schrieb Sage Weil:
> > > > On Sun, 4 Nov 2012, Stefan Priebe wrote:
> > > > > Can i merge wip-rbd-read into master?
> > > > 
> > > > Yeah.  I'm going to do a bit more testing first before I do it, but it
> > > > should apply cleanly.  Hopefully later today.
> > > 
> > > Any other branches to merge locally while benchmarking / testing?
> > 
> > Nope, although you get the latest wip-rbd-read (I fixed an issue with
> > glibc's random_r sucking).
> 
> Did i get you wrong? There's no commit from sage today?
> https://github.com/ceph/ceph/commits/wip-rbd-read

That's the right one.

(There was briefly some other stuff in wip-rbd-read earlier today that was 
broken.)

sage

Re: BUG: kvm crashing in void librbd::AioCompletion::complete_request

[Stefan Priebe]

Am 04.11.2012 15:12, schrieb Sage Weil:
> On Sun, 4 Nov 2012, Stefan Priebe wrote:
>> Can i merge wip-rbd-read into master?
>
> Yeah.  I'm going to do a bit more testing first before I do it, but it
> should apply cleanly.  Hopefully later today.

Thanks - seems to be fixed with wip-rbd-read but i have a memory leak now.

kvm process raises and raises and raises which each new test around 5GB 
of memory. Should i write a new mail?

Greets,
Stefan

Re: BUG: kvm crashing in void librbd::AioCompletion::complete_request

[Stefan Priebe]

Am 05.11.2012 20:33, schrieb Stefan Priebe:
> Am 04.11.2012 15:12, schrieb Sage Weil:
>> On Sun, 4 Nov 2012, Stefan Priebe wrote:
>>> Can i merge wip-rbd-read into master?
>>
>> Yeah.  I'm going to do a bit more testing first before I do it, but it
>> should apply cleanly.  Hopefully later today.
>
> Thanks - seems to be fixed with wip-rbd-read but i have a memory leak now.
>
> kvm process raises and raises and raises which each new test around 5GB
> of memory. Should i write a new mail?

Works fine with vanilla origin/master. So it must be in wip-rbd-read.

Stefan

Re: BUG: kvm crashing in void librbd::AioCompletion::complete_request

[Andrey Korolyov]

On Mon, Nov 5, 2012 at 11:33 PM, Stefan Priebe <s.priebe@profihost.ag> wrote:
> Am 04.11.2012 15:12, schrieb Sage Weil:
>>
>> On Sun, 4 Nov 2012, Stefan Priebe wrote:
>>>
>>> Can i merge wip-rbd-read into master?
>>
>>
>> Yeah.  I'm going to do a bit more testing first before I do it, but it
>> should apply cleanly.  Hopefully later today.
>
>
> Thanks - seems to be fixed with wip-rbd-read but i have a memory leak now.
>
> kvm process raises and raises and raises which each new test around 5GB of
> memory. Should i write a new mail?

Do you use qemu 1.2.0? It has a memory leak, at least with rbd
backend, but I didn`t profiled it yet, so it may be something generic
:)


> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: BUG: kvm crashing in void librbd::AioCompletion::complete_request

[Stefan Priebe]

Am 05.11.2012 21:20, schrieb Andrey Korolyov:
> On Mon, Nov 5, 2012 at 11:33 PM, Stefan Priebe <s.priebe@profihost.ag> wrote:
>> Am 04.11.2012 15:12, schrieb Sage Weil:
>>>
>>> On Sun, 4 Nov 2012, Stefan Priebe wrote:
>>>>
>>>> Can i merge wip-rbd-read into master?
>>>
>>>
>>> Yeah.  I'm going to do a bit more testing first before I do it, but it
>>> should apply cleanly.  Hopefully later today.
>>
>>
>> Thanks - seems to be fixed with wip-rbd-read but i have a memory leak now.
>>
>> kvm process raises and raises and raises which each new test around 5GB of
>> memory. Should i write a new mail?
>
> Do you use qemu 1.2.0? It has a memory leak, at least with rbd
> backend, but I didn`t profiled it yet, so it may be something generic
> :)

Yes i'm using kvm-qemu 1.2. But i dought that in my case it is a qemu 
problem. Memory usage is constant using ceph git master. When i merge 
ceph wip-rbd-read into master kvm process raises massivly while reading 
from disk.

Greets,
Stefan

Re: BUG: kvm crashing in void librbd::AioCompletion::complete_request

[Stefan Priebe]

Am 05.11.2012 20:33, schrieb Stefan Priebe:
> Am 04.11.2012 15:12, schrieb Sage Weil:
>> On Sun, 4 Nov 2012, Stefan Priebe wrote:
>>> Can i merge wip-rbd-read into master?
>>
>> Yeah.  I'm going to do a bit more testing first before I do it, but it
>> should apply cleanly.  Hopefully later today.
>
> Thanks - seems to be fixed with wip-rbd-read but i have a memory leak now.
>
> kvm process raises and raises and raises which each new test around 5GB
> of memory. Should i write a new mail?

Most mem get used while doing a lot of small reads.

Stefan

Re: BUG: kvm crashing in void librbd::AioCompletion::complete_request

[Alexandre DERUMIER]

>>Most mem get used while doing a lot of small reads. 

I see same problem with my test and wip-rbd-read.

I got also strange fio results, from 5k to 100K iops (increase and increase in time during the bench).




----- Mail original ----- 

De: "Stefan Priebe" <s.priebe@profihost.ag> 
À: "Sage Weil" <sage@inktank.com> 
Cc: ceph-devel@vger.kernel.org 
Envoyé: Lundi 5 Novembre 2012 21:43:05 
Objet: Re: BUG: kvm crashing in void librbd::AioCompletion::complete_request 

Am 05.11.2012 20:33, schrieb Stefan Priebe: 
> Am 04.11.2012 15:12, schrieb Sage Weil: 
>> On Sun, 4 Nov 2012, Stefan Priebe wrote: 
>>> Can i merge wip-rbd-read into master? 
>> 
>> Yeah. I'm going to do a bit more testing first before I do it, but it 
>> should apply cleanly. Hopefully later today. 
> 
> Thanks - seems to be fixed with wip-rbd-read but i have a memory leak now. 
> 
> kvm process raises and raises and raises which each new test around 5GB 
> of memory. Should i write a new mail? 

Most mem get used while doing a lot of small reads. 

Stefan 
-- 
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
the body of a message to majordomo@vger.kernel.org 
More majordomo info at http://vger.kernel.org/majordomo-info.html 
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: BUG: kvm crashing in void librbd::AioCompletion::complete_request

[Stefan Priebe - Profihost AG]

Am 06.11.2012 10:07, schrieb Alexandre DERUMIER:
>>> Most mem get used while doing a lot of small reads.
>
> I see same problem with my test and wip-rbd-read.
>
> I got also strange fio results, from 5k to 100K iops (increase and increase in time during the bench).

Update to latest wip-rbd-read frmo yesterday. Sage fixed the bug but 
didn't report back.

Stefan