RBD stall when creating and listing images

RBD stall when creating and listing images

[Wido den Hollander]

Hi,

I was just playing with my libvirt driver and I noticed a stall.

On one host I had running:

for i in {0..8192}; do rbd create --size 64 image-$i; echo "created 
image-$i"; done

And on the other host I was running a pool-refresh in libvirt.

As I was tailing the libvirt log I noticed that it had stalled, so I 
checked my other terminal and saw that the creation of new images had 
also stalled.

This situation persisted for about ~20 seconds and then everything 
started to continue.

At the moment of the stall libvirt was calling rbd_list and was not 
doing any rbd_stat yet.

I'm trying to reproduce it here, but I wonder if there is a logical 
explanation for this situation?

I'm running 0.44 on my cluster and 0.45 on the client btw.

Wido

Re: RBD stall when creating and listing images

[Sage Weil]

On Sat, 21 Apr 2012, Wido den Hollander wrote:
> Hi,
> 
> I was just playing with my libvirt driver and I noticed a stall.
> 
> On one host I had running:
> 
> for i in {0..8192}; do rbd create --size 64 image-$i; echo "created image-$i";
> done
> 
> And on the other host I was running a pool-refresh in libvirt.
> 
> As I was tailing the libvirt log I noticed that it had stalled, so I checked
> my other terminal and saw that the creation of new images had also stalled.
> 
> This situation persisted for about ~20 seconds and then everything started to
> continue.
> 
> At the moment of the stall libvirt was calling rbd_list and was not doing any
> rbd_stat yet.
> 
> I'm trying to reproduce it here, but I wonder if there is a logical
> explanation for this situation?
> 
> I'm running 0.44 on my cluster and 0.45 on the client btw.

Nothing comes to mind.. unless libvirt is doing something silly on 
pool-refresh and actually modifies each image header (which would make 
users stop and refresh).

If you can reproduce this with some logs (ideally the stalling client), 
I'd love to see what's going on.

sage

Re: RBD stall when creating and listing images

[Wido den Hollander]

On 04/22/2012 06:34 AM, Sage Weil wrote:
> On Sat, 21 Apr 2012, Wido den Hollander wrote:
>> Hi,
>>
>> I was just playing with my libvirt driver and I noticed a stall.
>>
>> On one host I had running:
>>
>> for i in {0..8192}; do rbd create --size 64 image-$i; echo "created image-$i";
>> done
>>
>> And on the other host I was running a pool-refresh in libvirt.
>>
>> As I was tailing the libvirt log I noticed that it had stalled, so I checked
>> my other terminal and saw that the creation of new images had also stalled.
>>
>> This situation persisted for about ~20 seconds and then everything started to
>> continue.
>>
>> At the moment of the stall libvirt was calling rbd_list and was not doing any
>> rbd_stat yet.
>>
>> I'm trying to reproduce it here, but I wonder if there is a logical
>> explanation for this situation?
>>
>> I'm running 0.44 on my cluster and 0.45 on the client btw.
>
> Nothing comes to mind.. unless libvirt is doing something silly on
> pool-refresh and actually modifies each image header (which would make
> users stop and refresh).
>

No, it doesn't. This is the libvirt code: 
https://github.com/wido/libvirt/blob/rbd/src/storage/storage_backend_rbd.c#L312

> If you can reproduce this with some logs (ideally the stalling client),
> I'd love to see what's going on.

I think a "rbd ls" on one host and "rbd create" on another host should 
give the same result.

I'll try with that and see what comes out of it.

Wido

>
> sage