[Qemu-devel] qemu : rbd block driver internal snapshot and vm_stop is hanging forever

[Qemu-devel] qemu : rbd block driver internal snapshot and vm_stop is hanging forever

[Alexandre DERUMIER]

Hi,

with qemu (2.4.1), if I do an internal snapshot of an rbd device,
then I pause the vm with vm_stop,

the qemu process is hanging forever


monitor commands to reproduce:


# snapshot_blkdev_internal drive-virtio0 yoursnapname
# stop




I don't see this with qcow2 or sheepdog block driver for example.


Regards,

Alexandre

Re: [Qemu-devel] qemu : rbd block driver internal snapshot and vm_stop is hanging forever

[Alexandre DERUMIER]

Some other infos:

I can reproduce it too with manual snapshot with rbd command


#rbd --image myrbdvolume snap create --snap snap1

qemu monitor:

#stop


This is with ceph hammer 0.94.5.


in qemu vm_stop, the only thing related to block driver are

    bdrv_drain_all();
    ret = bdrv_flush_all();


----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "ceph-devel" <ceph-devel@vger.kernel.org>, "qemu-devel" <qemu-devel@nongnu.org>
Envoyé: Lundi 9 Novembre 2015 04:10:45
Objet: qemu : rbd block driver internal snapshot and vm_stop is hanging forever

Hi, 

with qemu (2.4.1), if I do an internal snapshot of an rbd device, 
then I pause the vm with vm_stop, 

the qemu process is hanging forever 


monitor commands to reproduce: 


# snapshot_blkdev_internal drive-virtio0 yoursnapname 
# stop 




I don't see this with qcow2 or sheepdog block driver for example. 


Regards, 

Alexandre 
-- 
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
the body of a message to majordomo@vger.kernel.org 
More majordomo info at http://vger.kernel.org/majordomo-info.html 

Re: [Qemu-devel] qemu : rbd block driver internal snapshot and vm_stop is hanging forever

[Alexandre DERUMIER]

Also, 

this occur only with rbd_cache=false or qemu drive cache=none.


If I use rbd_cache=true or qemu drive cache=writeback, I don't have this bug.


----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "ceph-devel" <ceph-devel@vger.kernel.org>, "qemu-devel" <qemu-devel@nongnu.org>
Envoyé: Lundi 9 Novembre 2015 04:23:10
Objet: Re: qemu : rbd block driver internal snapshot and vm_stop is hanging forever

Some other infos: 

I can reproduce it too with manual snapshot with rbd command 


#rbd --image myrbdvolume snap create --snap snap1 

qemu monitor: 

#stop 


This is with ceph hammer 0.94.5. 


in qemu vm_stop, the only thing related to block driver are 

bdrv_drain_all(); 
ret = bdrv_flush_all(); 


----- Mail original ----- 
De: "aderumier" <aderumier@odiso.com> 
À: "ceph-devel" <ceph-devel@vger.kernel.org>, "qemu-devel" <qemu-devel@nongnu.org> 
Envoyé: Lundi 9 Novembre 2015 04:10:45 
Objet: qemu : rbd block driver internal snapshot and vm_stop is hanging forever 

Hi, 

with qemu (2.4.1), if I do an internal snapshot of an rbd device, 
then I pause the vm with vm_stop, 

the qemu process is hanging forever 


monitor commands to reproduce: 


# snapshot_blkdev_internal drive-virtio0 yoursnapname 
# stop 




I don't see this with qcow2 or sheepdog block driver for example. 


Regards, 

Alexandre 
-- 
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
the body of a message to majordomo@vger.kernel.org 
More majordomo info at http://vger.kernel.org/majordomo-info.html 

-- 
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
the body of a message to majordomo@vger.kernel.org 
More majordomo info at http://vger.kernel.org/majordomo-info.html 

Re: [Qemu-devel] qemu : rbd block driver internal snapshot and vm_stop is hanging forever

[Alexandre DERUMIER]

Something is really wrong,

because guest is also freezing, with a simple snapshot, with cache=none / rbd_cache=false

qemu monitor : snapshot_blkdev_internal drive-virtio0 snap1

or 

rbd command : rbd --image myrbdvolume snap create --snap snap1 


Then the guest can't read/write to disk anymore.


I have tested with last ceph internalis, same problem



----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "ceph-devel" <ceph-devel@vger.kernel.org>, "qemu-devel" <qemu-devel@nongnu.org>
Envoyé: Lundi 9 Novembre 2015 04:54:23
Objet: Re: [Qemu-devel] qemu : rbd block driver internal snapshot and vm_stop is hanging forever

Also, 

this occur only with rbd_cache=false or qemu drive cache=none. 


If I use rbd_cache=true or qemu drive cache=writeback, I don't have this bug. 


----- Mail original ----- 
De: "aderumier" <aderumier@odiso.com> 
À: "ceph-devel" <ceph-devel@vger.kernel.org>, "qemu-devel" <qemu-devel@nongnu.org> 
Envoyé: Lundi 9 Novembre 2015 04:23:10 
Objet: Re: qemu : rbd block driver internal snapshot and vm_stop is hanging forever 

Some other infos: 

I can reproduce it too with manual snapshot with rbd command 


#rbd --image myrbdvolume snap create --snap snap1 

qemu monitor: 

#stop 


This is with ceph hammer 0.94.5. 


in qemu vm_stop, the only thing related to block driver are 

bdrv_drain_all(); 
ret = bdrv_flush_all(); 


----- Mail original ----- 
De: "aderumier" <aderumier@odiso.com> 
À: "ceph-devel" <ceph-devel@vger.kernel.org>, "qemu-devel" <qemu-devel@nongnu.org> 
Envoyé: Lundi 9 Novembre 2015 04:10:45 
Objet: qemu : rbd block driver internal snapshot and vm_stop is hanging forever 

Hi, 

with qemu (2.4.1), if I do an internal snapshot of an rbd device, 
then I pause the vm with vm_stop, 

the qemu process is hanging forever 


monitor commands to reproduce: 


# snapshot_blkdev_internal drive-virtio0 yoursnapname 
# stop 




I don't see this with qcow2 or sheepdog block driver for example. 


Regards, 

Alexandre 
-- 
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
the body of a message to majordomo@vger.kernel.org 
More majordomo info at http://vger.kernel.org/majordomo-info.html 

-- 
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
the body of a message to majordomo@vger.kernel.org 
More majordomo info at http://vger.kernel.org/majordomo-info.html 

Re: [Qemu-devel] qemu : rbd block driver internal snapshot and vm_stop is hanging forever

[Denis V. Lunev]

On 11/09/2015 06:10 AM, Alexandre DERUMIER wrote:
> Hi,
>
> with qemu (2.4.1), if I do an internal snapshot of an rbd device,
> then I pause the vm with vm_stop,
>
> the qemu process is hanging forever
>
>
> monitor commands to reproduce:
>
>
> # snapshot_blkdev_internal drive-virtio0 yoursnapname
> # stop
>
>
>
>
> I don't see this with qcow2 or sheepdog block driver for example.
>
>
> Regards,
>
> Alexandre
>
this could look like the problem I have recenty trying to
fix with dataplane enabled. Patch series is named as

[PATCH for 2.5 v6 0/10] dataplane snapshot fixes

Den

Re: [Qemu-devel] qemu : rbd block driver internal snapshot and vm_stop is hanging forever

[Denis V. Lunev]

On 11/09/2015 10:19 AM, Denis V. Lunev wrote:
> On 11/09/2015 06:10 AM, Alexandre DERUMIER wrote:
>> Hi,
>>
>> with qemu (2.4.1), if I do an internal snapshot of an rbd device,
>> then I pause the vm with vm_stop,
>>
>> the qemu process is hanging forever
>>
>>
>> monitor commands to reproduce:
>>
>>
>> # snapshot_blkdev_internal drive-virtio0 yoursnapname
>> # stop
>>
>>
>>
>>
>> I don't see this with qcow2 or sheepdog block driver for example.
>>
>>
>> Regards,
>>
>> Alexandre
>>
> this could look like the problem I have recenty trying to
> fix with dataplane enabled. Patch series is named as
>
> [PATCH for 2.5 v6 0/10] dataplane snapshot fixes
>
> Den

anyway, even if above will not help, can you collect gdb
traces from all threads in QEMU process. May be I'll be
able to give a hit.

Den

Re: [Qemu-devel] qemu : rbd block driver internal snapshot and vm_stop is hanging forever

[Alexandre DERUMIER]

adding to ceph.conf

[client]
        rbd_non_blocking_aio = false


fix the problem for me (with rbd_cache=false)


(@cc jdurgin@redhat.com)



----- Mail original -----
De: "Denis V. Lunev" <den-lists@parallels.com>
À: "aderumier" <aderumier@odiso.com>, "ceph-devel" <ceph-devel@vger.kernel.org>, "qemu-devel" <qemu-devel@nongnu.org>
Envoyé: Lundi 9 Novembre 2015 08:22:34
Objet: Re: [Qemu-devel] qemu : rbd block driver internal snapshot and vm_stop is hanging forever

On 11/09/2015 10:19 AM, Denis V. Lunev wrote: 
> On 11/09/2015 06:10 AM, Alexandre DERUMIER wrote: 
>> Hi, 
>> 
>> with qemu (2.4.1), if I do an internal snapshot of an rbd device, 
>> then I pause the vm with vm_stop, 
>> 
>> the qemu process is hanging forever 
>> 
>> 
>> monitor commands to reproduce: 
>> 
>> 
>> # snapshot_blkdev_internal drive-virtio0 yoursnapname 
>> # stop 
>> 
>> 
>> 
>> 
>> I don't see this with qcow2 or sheepdog block driver for example. 
>> 
>> 
>> Regards, 
>> 
>> Alexandre 
>> 
> this could look like the problem I have recenty trying to 
> fix with dataplane enabled. Patch series is named as 
> 
> [PATCH for 2.5 v6 0/10] dataplane snapshot fixes 
> 
> Den 

anyway, even if above will not help, can you collect gdb 
traces from all threads in QEMU process. May be I'll be 
able to give a hit. 

Den 

Re: [Qemu-devel] qemu : rbd block driver internal snapshot and vm_stop is hanging forever

[Jason Dillaman]

Can you reproduce with Ceph debug logging enabled (i.e. debug rbd=20 in your ceph.conf)?  If you could attach the log to the Ceph tracker ticket I opened [1], that would be very helpful.

[1] http://tracker.ceph.com/issues/13726

Thanks,
Jason 


----- Original Message -----
> From: "Alexandre DERUMIER" <aderumier@odiso.com>
> To: "ceph-devel" <ceph-devel@vger.kernel.org>
> Cc: "qemu-devel" <qemu-devel@nongnu.org>, jdurgin@redhat.com
> Sent: Monday, November 9, 2015 5:48:45 AM
> Subject: Re: [Qemu-devel] qemu : rbd block driver internal snapshot and vm_stop is hanging forever
> 
> adding to ceph.conf
> 
> [client]
>         rbd_non_blocking_aio = false
> 
> 
> fix the problem for me (with rbd_cache=false)
> 
> 
> (@cc jdurgin@redhat.com)
> 
> 
> 
> ----- Mail original -----
> De: "Denis V. Lunev" <den-lists@parallels.com>
> À: "aderumier" <aderumier@odiso.com>, "ceph-devel"
> <ceph-devel@vger.kernel.org>, "qemu-devel" <qemu-devel@nongnu.org>
> Envoyé: Lundi 9 Novembre 2015 08:22:34
> Objet: Re: [Qemu-devel] qemu : rbd block driver internal snapshot and vm_stop
> is hanging forever
> 
> On 11/09/2015 10:19 AM, Denis V. Lunev wrote:
> > On 11/09/2015 06:10 AM, Alexandre DERUMIER wrote:
> >> Hi,
> >> 
> >> with qemu (2.4.1), if I do an internal snapshot of an rbd device,
> >> then I pause the vm with vm_stop,
> >> 
> >> the qemu process is hanging forever
> >> 
> >> 
> >> monitor commands to reproduce:
> >> 
> >> 
> >> # snapshot_blkdev_internal drive-virtio0 yoursnapname
> >> # stop
> >> 
> >> 
> >> 
> >> 
> >> I don't see this with qcow2 or sheepdog block driver for example.
> >> 
> >> 
> >> Regards,
> >> 
> >> Alexandre
> >> 
> > this could look like the problem I have recenty trying to
> > fix with dataplane enabled. Patch series is named as
> > 
> > [PATCH for 2.5 v6 0/10] dataplane snapshot fixes
> > 
> > Den
> 
> anyway, even if above will not help, can you collect gdb
> traces from all threads in QEMU process. May be I'll be
> able to give a hit.
> 
> Den
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: [Qemu-devel] qemu : rbd block driver internal snapshot and vm_stop is hanging forever

[Stefan Priebe - Profihost AG]

> ----- Original Message -----
>> From: "Alexandre DERUMIER" <aderumier@odiso.com>
>> To: "ceph-devel" <ceph-devel@vger.kernel.org>
>> Cc: "qemu-devel" <qemu-devel@nongnu.org>, jdurgin@redhat.com
>> Sent: Monday, November 9, 2015 5:48:45 AM
>> Subject: Re: [Qemu-devel] qemu : rbd block driver internal snapshot and vm_stop is hanging forever
>>
>> adding to ceph.conf
>>
>> [client]
>>         rbd_non_blocking_aio = false
>>
>>
>> fix the problem for me (with rbd_cache=false)
>>
>>
>> (@cc jdurgin@redhat.com)

+1 same to me.

Stefan

>>
>>
>>
>> ----- Mail original -----
>> De: "Denis V. Lunev" <den-lists@parallels.com>
>> À: "aderumier" <aderumier@odiso.com>, "ceph-devel"
>> <ceph-devel@vger.kernel.org>, "qemu-devel" <qemu-devel@nongnu.org>
>> Envoyé: Lundi 9 Novembre 2015 08:22:34
>> Objet: Re: [Qemu-devel] qemu : rbd block driver internal snapshot and vm_stop
>> is hanging forever
>>
>> On 11/09/2015 10:19 AM, Denis V. Lunev wrote:
>>> On 11/09/2015 06:10 AM, Alexandre DERUMIER wrote:
>>>> Hi,
>>>>
>>>> with qemu (2.4.1), if I do an internal snapshot of an rbd device,
>>>> then I pause the vm with vm_stop,
>>>>
>>>> the qemu process is hanging forever
>>>>
>>>>
>>>> monitor commands to reproduce:
>>>>
>>>>
>>>> # snapshot_blkdev_internal drive-virtio0 yoursnapname
>>>> # stop
>>>>
>>>>
>>>>
>>>>
>>>> I don't see this with qcow2 or sheepdog block driver for example.
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Alexandre
>>>>
>>> this could look like the problem I have recenty trying to
>>> fix with dataplane enabled. Patch series is named as
>>>
>>> [PATCH for 2.5 v6 0/10] dataplane snapshot fixes
>>>
>>> Den
>>
>> anyway, even if above will not help, can you collect gdb
>> traces from all threads in QEMU process. May be I'll be
>> able to give a hit.
>>
>> Den
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: [Qemu-devel] qemu : rbd block driver internal snapshot and vm_stop is hanging forever

[Alexandre DERUMIER]

>>Can you reproduce with Ceph debug logging enabled (i.e. debug rbd=20 in your ceph.conf)?  If you could attach the log to the Ceph tracker ticket I opened [1], that would be very helpful.
>>
>>[1] http://tracker.ceph.com/issues/13726

yes,I'm able to reproduce it 100%, I have attached the log to the tracker.

Alexandre

----- Mail original -----
De: "Jason Dillaman" <dillaman@redhat.com>
À: "aderumier" <aderumier@odiso.com>
Cc: "ceph-devel" <ceph-devel@vger.kernel.org>, "qemu-devel" <qemu-devel@nongnu.org>, jdurgin@redhat.com
Envoyé: Lundi 9 Novembre 2015 14:42:42
Objet: Re: [Qemu-devel] qemu : rbd block driver internal snapshot and vm_stop is hanging forever

Can you reproduce with Ceph debug logging enabled (i.e. debug rbd=20 in your ceph.conf)? If you could attach the log to the Ceph tracker ticket I opened [1], that would be very helpful. 

[1] http://tracker.ceph.com/issues/13726 

Thanks, 
Jason 


----- Original Message ----- 
> From: "Alexandre DERUMIER" <aderumier@odiso.com> 
> To: "ceph-devel" <ceph-devel@vger.kernel.org> 
> Cc: "qemu-devel" <qemu-devel@nongnu.org>, jdurgin@redhat.com 
> Sent: Monday, November 9, 2015 5:48:45 AM 
> Subject: Re: [Qemu-devel] qemu : rbd block driver internal snapshot and vm_stop is hanging forever 
> 
> adding to ceph.conf 
> 
> [client] 
> rbd_non_blocking_aio = false 
> 
> 
> fix the problem for me (with rbd_cache=false) 
> 
> 
> (@cc jdurgin@redhat.com) 
> 
> 
> 
> ----- Mail original ----- 
> De: "Denis V. Lunev" <den-lists@parallels.com> 
> À: "aderumier" <aderumier@odiso.com>, "ceph-devel" 
> <ceph-devel@vger.kernel.org>, "qemu-devel" <qemu-devel@nongnu.org> 
> Envoyé: Lundi 9 Novembre 2015 08:22:34 
> Objet: Re: [Qemu-devel] qemu : rbd block driver internal snapshot and vm_stop 
> is hanging forever 
> 
> On 11/09/2015 10:19 AM, Denis V. Lunev wrote: 
> > On 11/09/2015 06:10 AM, Alexandre DERUMIER wrote: 
> >> Hi, 
> >> 
> >> with qemu (2.4.1), if I do an internal snapshot of an rbd device, 
> >> then I pause the vm with vm_stop, 
> >> 
> >> the qemu process is hanging forever 
> >> 
> >> 
> >> monitor commands to reproduce: 
> >> 
> >> 
> >> # snapshot_blkdev_internal drive-virtio0 yoursnapname 
> >> # stop 
> >> 
> >> 
> >> 
> >> 
> >> I don't see this with qcow2 or sheepdog block driver for example. 
> >> 
> >> 
> >> Regards, 
> >> 
> >> Alexandre 
> >> 
> > this could look like the problem I have recenty trying to 
> > fix with dataplane enabled. Patch series is named as 
> > 
> > [PATCH for 2.5 v6 0/10] dataplane snapshot fixes 
> > 
> > Den 
> 
> anyway, even if above will not help, can you collect gdb 
> traces from all threads in QEMU process. May be I'll be 
> able to give a hit. 
> 
> Den 
> -- 
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
> the body of a message to majordomo@vger.kernel.org 
> More majordomo info at http://vger.kernel.org/majordomo-info.html 
>