Errors attaching RBD image to a running VM

Errors attaching RBD image to a running VM

[Michael Morgan]

Hello,

 I've been building an OpenStack cluster using Ceph as the storage backend.
Currently running SL 6.3, OpenStack Folsom packages from EPEL, Libvirt 0.9.10,
QEMU 1.2.1 built from an epel-testing SRPM for RBD support, and Ceph 0.54. I
can boot off of an image without problems but attaching to a running instance
fails. Tracking back from Horizon and Cinder I see Libvirt generating the
errors:

2012-12-07 20:03:20.657+0000: 19803: error : qemuMonitorJSONCheckError:338 : internal error unable to execute QEMU command 'device_add': Property 'virtio-blk-pci.drive' can't find value 'drive-virtio-disk1'
2012-12-07 20:03:20.664+0000: 19803: error : qemuMonitorTextDriveDel:2895 : operation failed: deleting file=rbd:volumes/volume-6c82e5d3-e697-43a8-8194-1f7df932ceb8:id=volumes:key=AQCicbZQ8Oo2IBAALganQv+zY/jjECc9fHUBBA==:auth_supported=cephx none,if=none,id=drive-virtio-disk1,format=raw,cache=none drive failed: drive_del: extraneous characters at the end of line

 I can reproduce the error with virsh attach-device and the following XML:

<disk type='network' device='disk'>
  <driver name='qemu' type='raw' cache='none'/>
  <auth username='volumes'>
    <secret type='ceph' uuid='93fb3d32-7e2d-691d-db94-4c1cf21bed02'/>
  </auth>
  <source protocol="rbd" name="volumes/volume-6c82e5d3-e697-43a8-8194-1f7df932ceb8"/>
  <target bus='virtio' dev='vdb'/>
</disk>
 
 I realize this is probably more of a Libvirt/QEMU problem but the only
reference I could find to an error like this was a post here from last year:

http://permalink.gmane.org/gmane.comp.file-systems.ceph.devel/4713

 In that thread the problem disappeared after multiple version changes without
any clear source for the resolution. Has anyone seen this type of error before?

 I can certainly move along to other lists if appropriate but figured I'd start
here since it's the only place I saw such an issue pop up. It also gives me an
opportunity to say how awesome Ceph seems. The level of support I've seen on
this list is pretty amazing and I hope to increase our use of Ceph in the
future.

Thanks in advance,
 Mike

Re: Errors attaching RBD image to a running VM

[Josh Durgin]

On 12/07/2012 01:22 PM, Michael Morgan wrote:
> Hello,
>
>   I've been building an OpenStack cluster using Ceph as the storage backend.
> Currently running SL 6.3, OpenStack Folsom packages from EPEL, Libvirt 0.9.10,
> QEMU 1.2.1 built from an epel-testing SRPM for RBD support, and Ceph 0.54. I
> can boot off of an image without problems but attaching to a running instance
> fails. Tracking back from Horizon and Cinder I see Libvirt generating the
> errors:
>
> 2012-12-07 20:03:20.657+0000: 19803: error : qemuMonitorJSONCheckError:338 : internal error unable to execute QEMU command 'device_add': Property 'virtio-blk-pci.drive' can't find value 'drive-virtio-disk1'
> 2012-12-07 20:03:20.664+0000: 19803: error : qemuMonitorTextDriveDel:2895 : operation failed: deleting file=rbd:volumes/volume-6c82e5d3-e697-43a8-8194-1f7df932ceb8:id=volumes:key=AQCicbZQ8Oo2IBAALganQv+zY/jjECc9fHUBBA==:auth_supported=cephx none,if=none,id=drive-virtio-disk1,format=raw,cache=none drive failed: drive_del: extraneous characters at the end of line
>
>   I can reproduce the error with virsh attach-device and the following XML:
>
> <disk type='network' device='disk'>
>    <driver name='qemu' type='raw' cache='none'/>
>    <auth username='volumes'>
>      <secret type='ceph' uuid='93fb3d32-7e2d-691d-db94-4c1cf21bed02'/>
>    </auth>
>    <source protocol="rbd" name="volumes/volume-6c82e5d3-e697-43a8-8194-1f7df932ceb8"/>
>    <target bus='virtio' dev='vdb'/>
> </disk>
>
>   I realize this is probably more of a Libvirt/QEMU problem but the only
> reference I could find to an error like this was a post here from last year:
>
> http://permalink.gmane.org/gmane.comp.file-systems.ceph.devel/4713
>
>   In that thread the problem disappeared after multiple version changes without
> any clear source for the resolution. Has anyone seen this type of error before?

Yes, this is fixed in libvirt 0.9.12. There were a couple issues with
properly escaping JSON and using a space instead of a semicolon with
qemu's command parsing.

There are a couple more rbd-related bug fixes in later versions of
libvirt, but you wouldn't hit those problems using OpenStack.

>   I can certainly move along to other lists if appropriate but figured I'd start
> here since it's the only place I saw such an issue pop up. It also gives me an
> opportunity to say how awesome Ceph seems. The level of support I've seen on
> this list is pretty amazing and I hope to increase our use of Ceph in the
> future.

Thanks :). I actually prefer ceph-related stuff coming to this list,
since it's easier to keep track of and gets more attention from people
more familiar with ceph, rather than just libvirt, qemu, etc.

Josh

Re: Errors attaching RBD image to a running VM

[Michael Morgan]

On Fri, Dec 07, 2012 at 02:50:08PM -0800, Josh Durgin wrote:
> Yes, this is fixed in libvirt 0.9.12. There were a couple issues with
> properly escaping JSON and using a space instead of a semicolon with
> qemu's command parsing.
> 
> There are a couple more rbd-related bug fixes in later versions of
> libvirt, but you wouldn't hit those problems using OpenStack.

Thanks Josh! After building and installing libvirt 1.0.0 RPMs I unfortunately
seem to have taken a step backwards. Any use of network disks with
protocol="rbd" causes libvirt to bail with a dreaded 'No such file or
directory':

2012-12-10 20:08:37.203+0000: 16691: error : virFileOpenAs:1077 : failed to open file 'volumes/volume-485390a9-5a39-46c1-8434-872acbc645d8': No such file or directory
2012-12-10 20:08:37.203+0000: 16691: error : virStorageFileGetMetadataRecurse:962 : cannot open file 'volumes/volume-485390a9-5a39-46c1-8434-872acbc645d8': No such file or directory

I see there is a specific flag in the spec file for RBD support in this version
but I'm assuming that's more for pool support, correct? I get the same failures
either way. Is my disk XML now wrong with newer versions? Or perhaps is there a
recommended version of libvirt besides the latest 1.0.0?

I think I'm so close. Attaching volumes is pretty much the last piece of
functionality before I start just hammering this platform for stability. Thanks
again!

-Mike

--
Michael Morgan
Network Technician, DCANet
http://www.dca.net
mmorgan@dca.net
(888) 432-2638 x4723

Re: Errors attaching RBD image to a running VM

[Josh Durgin]

On 12/10/2012 12:48 PM, Michael Morgan wrote:
> On Fri, Dec 07, 2012 at 02:50:08PM -0800, Josh Durgin wrote:
>> Yes, this is fixed in libvirt 0.9.12. There were a couple issues with
>> properly escaping JSON and using a space instead of a semicolon with
>> qemu's command parsing.
>>
>> There are a couple more rbd-related bug fixes in later versions of
>> libvirt, but you wouldn't hit those problems using OpenStack.
>
> Thanks Josh! After building and installing libvirt 1.0.0 RPMs I unfortunately
> seem to have taken a step backwards. Any use of network disks with
> protocol="rbd" causes libvirt to bail with a dreaded 'No such file or
> directory':
>
> 2012-12-10 20:08:37.203+0000: 16691: error : virFileOpenAs:1077 : failed to open file 'volumes/volume-485390a9-5a39-46c1-8434-872acbc645d8': No such file or directory
> 2012-12-10 20:08:37.203+0000: 16691: error : virStorageFileGetMetadataRecurse:962 : cannot open file 'volumes/volume-485390a9-5a39-46c1-8434-872acbc645d8': No such file or directory

There was a regression in 1.0.0 with attaching non-files, such as RBD.
This is fixed by f0e72b2f5c675f927d04545dc5095f9e5998f171, which you
could cherry-pick onto 1.0.0.

If you'd rather just use a released version, 0.10.2 should be fine.

> I see there is a specific flag in the spec file for RBD support in this version
> but I'm assuming that's more for pool support, correct?

Yes, --with-storage-rbd is just for using rbd as a backend for libvirt
pools.

Josh

Re: Errors attaching RBD image to a running VM

[Michael Morgan]

On Mon, Dec 10, 2012 at 01:12:45PM -0800, Josh Durgin wrote:
> 
> There was a regression in 1.0.0 with attaching non-files, such as RBD.
> This is fixed by f0e72b2f5c675f927d04545dc5095f9e5998f171, which you
> could cherry-pick onto 1.0.0.
> 
> If you'd rather just use a released version, 0.10.2 should be fine.
> 

Libvirt 0.10.2 appears to have fixed my problems. Booting from RBD works as
well as attaching/detaching multiple additional volumes. Thanks so much Josh!

-Mike

Re: Errors attaching RBD image to a running VM

[venkat]

same issue with libvirt version 1.2.9,
can somebody help me.

Thanks & Regards,
Venkat 

Re: Errors attaching RBD image to a running VM

[venkat]

 same issue with libvirt version 1.2.9,
 can somebody help me.

 Thanks & Regards,
 Venkat 

Re: Errors attaching RBD image to a running VM

[Shinobu Kinjo]

Hello Venkat,

Thank you!!

 Shinobu

----- Original Message -----
From: "B, Naga Venkata" <naga.b@hpe.com>
To: skinjo@redhat.com
Cc: "Ceph Development" <ceph-devel@vger.kernel.org>
Sent: Wednesday, August 26, 2015 7:24:17 PM
Subject: RE: Errors attaching RBD image to a running VM

Hi Shinobu,

This is from http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/11230.

Thanks & Regards,
Venkat


From: Shinobu [mailto:shinobu.kj@gmail.com]
Sent: Tuesday, August 25, 2015 5:35 PM
To: B, Naga Venkata
Cc: Ceph Development
Subject: Re: Errors attaching RBD image to a running VM

Where is original message?
Shinobu

On Tue, Aug 25, 2015 at 8:31 PM, venkat <naga.b@hp.com<mailto:naga.b@hp.com>> wrote:



same issue with libvirt version 1.2.9,
can somebody help me.

Thanks & Regards,
Venkat

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org<mailto:majordomo@vger.kernel.org>
More majordomo info at  http://vger.kernel.org/majordomo-info.html



--
Email:
 shinobu@linux.com<mailto:shinobu@linux.com>
 skinjo@redhat.com<mailto:skinjo@redhat.com>

 Life w/ Linux<http://i-shinobu.hatenablog.com/>

RE: Errors attaching RBD image to a running VM

[B, Naga Venkata]

Hi Shinobu,

Can I have quick reply on this, is this issue with libvirt version or ceph.

Thanks & Regards,
Venkat

-----Original Message-----
From: Shinobu Kinjo [mailto:skinjo@redhat.com] 
Sent: Wednesday, August 26, 2015 5:54 PM
To: B, Naga Venkata
Cc: Ceph Development
Subject: Re: Errors attaching RBD image to a running VM

Hello Venkat,

Thank you!!

 Shinobu

----- Original Message -----
From: "B, Naga Venkata" <naga.b@hpe.com>
To: skinjo@redhat.com
Cc: "Ceph Development" <ceph-devel@vger.kernel.org>
Sent: Wednesday, August 26, 2015 7:24:17 PM
Subject: RE: Errors attaching RBD image to a running VM

Hi Shinobu,

This is from http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/11230.

Thanks & Regards,
Venkat


From: Shinobu [mailto:shinobu.kj@gmail.com]
Sent: Tuesday, August 25, 2015 5:35 PM
To: B, Naga Venkata
Cc: Ceph Development
Subject: Re: Errors attaching RBD image to a running VM

Where is original message?
Shinobu

On Tue, Aug 25, 2015 at 8:31 PM, venkat <naga.b@hp.com<mailto:naga.b@hp.com>> wrote:



same issue with libvirt version 1.2.9,
can somebody help me.

Thanks & Regards,
Venkat

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org<mailto:majordomo@vger.kernel.org>
More majordomo info at  http://vger.kernel.org/majordomo-info.html



--
Email:
 shinobu@linux.com<mailto:shinobu@linux.com>
 skinjo@redhat.com<mailto:skinjo@redhat.com>

 Life w/ Linux<http://i-shinobu.hatenablog.com/>

Re: Errors attaching RBD image to a running VM

[Shinobu Kinjo]

Hello,

My question from first look at is that, is it your environment?

> Currently running SL 6.3, OpenStack Folsom packages from EPEL, Libvirt 0.9.10,
> QEMU 1.2.1 built from an epel-testing SRPM for RBD support, and Ceph 0.54

If it's true, you have to upgrade SL and Ceph as much as you can.
Because, as you might notice, there have been a bunch of bug fixes.

And please attach logs related to Ceph, qemu and livbirt.

Shinobu

----- Original Message -----
From: "B, Naga Venkata" <naga.b@hpe.com>
To: "Shinobu Kinjo" <skinjo@redhat.com>
Cc: "Ceph Development" <ceph-devel@vger.kernel.org>
Sent: Thursday, August 27, 2015 7:12:36 PM
Subject: RE: Errors attaching RBD image to a running VM

Hi Shinobu,

Can I have quick reply on this, is this issue with libvirt version or ceph.

Thanks & Regards,
Venkat

-----Original Message-----
From: Shinobu Kinjo [mailto:skinjo@redhat.com] 
Sent: Wednesday, August 26, 2015 5:54 PM
To: B, Naga Venkata
Cc: Ceph Development
Subject: Re: Errors attaching RBD image to a running VM

Hello Venkat,

Thank you!!

 Shinobu

----- Original Message -----
From: "B, Naga Venkata" <naga.b@hpe.com>
To: skinjo@redhat.com
Cc: "Ceph Development" <ceph-devel@vger.kernel.org>
Sent: Wednesday, August 26, 2015 7:24:17 PM
Subject: RE: Errors attaching RBD image to a running VM

Hi Shinobu,

This is from http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/11230.

Thanks & Regards,
Venkat


From: Shinobu [mailto:shinobu.kj@gmail.com]
Sent: Tuesday, August 25, 2015 5:35 PM
To: B, Naga Venkata
Cc: Ceph Development
Subject: Re: Errors attaching RBD image to a running VM

Where is original message?
Shinobu

On Tue, Aug 25, 2015 at 8:31 PM, venkat <naga.b@hp.com<mailto:naga.b@hp.com>> wrote:



same issue with libvirt version 1.2.9,
can somebody help me.

Thanks & Regards,
Venkat

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org<mailto:majordomo@vger.kernel.org>
More majordomo info at  http://vger.kernel.org/majordomo-info.html



--
Email:
 shinobu@linux.com<mailto:shinobu@linux.com>
 skinjo@redhat.com<mailto:skinjo@redhat.com>

 Life w/ Linux<http://i-shinobu.hatenablog.com/>

RE: Errors attaching RBD image to a running VM

[B, Naga Venkata]

NO, debian 3.14.44-1-amd64, QEMU emulator version 2.3.0, openstack kilo, libvirt 1.2.9, ceph version 0.80.7

I am to attach volume to nova instance if instance is in shutoff state. For that following steps I did.

I restarted libvirtd and libvirt-guests, after libvirt-guests restart vm went to shutoff state but nova is not updated instance state still it is in running state, now I did volume attach it went fine.
If I restart nova-compute instance state is updated with shutoff state, if I try to start instance the below error I am facing.
 
libvirtError: internal error: process exited while connecting to monitor: 2015-08-27T11:16:02.726265Z qemu-system-x86_64: -drive file=rbd:volumes/volume-ee7f6e3d-8269-4ffc-8d62-a881a0d7116f:id=cinder:key=AQCfEdxVUP13MBAAv4Oj7twFIxaF1BXyY1PkRQ==:auth_supported=cephx\;none:mon_host=192.168.245.2\:6789\;192.168.245.3\:6789\;192.168.245.4\:6789,if=none,id=drive-virtio-disk1,format=raw,serial=ee7f6e3d-8269-4ffc-8d62-a881a0d7116f,cache=none: Unknown protocol 'rbd'

Still this error "Errors attaching RBD image to a running VM" is there.

Thanks & Regards,
Venkat
-----Original Message-----
From: Shinobu Kinjo [mailto:skinjo@redhat.com] 
Sent: Thursday, August 27, 2015 6:00 PM
To: B, Naga Venkata
Cc: Ceph Development
Subject: Re: Errors attaching RBD image to a running VM

Hello,

My question from first look at is that, is it your environment?

> Currently running SL 6.3, OpenStack Folsom packages from EPEL, Libvirt 
> 0.9.10, QEMU 1.2.1 built from an epel-testing SRPM for RBD support, 
> and Ceph 0.54

If it's true, you have to upgrade SL and Ceph as much as you can.
Because, as you might notice, there have been a bunch of bug fixes.

And please attach logs related to Ceph, qemu and livbirt.

Shinobu

----- Original Message -----
From: "B, Naga Venkata" <naga.b@hpe.com>
To: "Shinobu Kinjo" <skinjo@redhat.com>
Cc: "Ceph Development" <ceph-devel@vger.kernel.org>
Sent: Thursday, August 27, 2015 7:12:36 PM
Subject: RE: Errors attaching RBD image to a running VM

Hi Shinobu,

Can I have quick reply on this, is this issue with libvirt version or ceph.

Thanks & Regards,
Venkat

-----Original Message-----
From: Shinobu Kinjo [mailto:skinjo@redhat.com]
Sent: Wednesday, August 26, 2015 5:54 PM
To: B, Naga Venkata
Cc: Ceph Development
Subject: Re: Errors attaching RBD image to a running VM

Hello Venkat,

Thank you!!

 Shinobu

----- Original Message -----
From: "B, Naga Venkata" <naga.b@hpe.com>
To: skinjo@redhat.com
Cc: "Ceph Development" <ceph-devel@vger.kernel.org>
Sent: Wednesday, August 26, 2015 7:24:17 PM
Subject: RE: Errors attaching RBD image to a running VM

Hi Shinobu,

This is from http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/11230.

Thanks & Regards,
Venkat


From: Shinobu [mailto:shinobu.kj@gmail.com]
Sent: Tuesday, August 25, 2015 5:35 PM
To: B, Naga Venkata
Cc: Ceph Development
Subject: Re: Errors attaching RBD image to a running VM

Where is original message?
Shinobu

On Tue, Aug 25, 2015 at 8:31 PM, venkat <naga.b@hp.com<mailto:naga.b@hp.com>> wrote:



same issue with libvirt version 1.2.9,
can somebody help me.

Thanks & Regards,
Venkat

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org<mailto:majordomo@vger.kernel.org>
More majordomo info at  http://vger.kernel.org/majordomo-info.html



--
Email:
 shinobu@linux.com<mailto:shinobu@linux.com>
 skinjo@redhat.com<mailto:skinjo@redhat.com>

 Life w/ Linux<http://i-shinobu.hatenablog.com/>

Re: Errors attaching RBD image to a running VM

[Shinobu Kinjo]

O.k, I got you -;

> NO, debian 3.14.44-1-amd64, QEMU emulator version 2.3.0, openstack kilo, libvirt 1.2.9, ceph version 0.80.7

How about this?

> And please attach logs related to Ceph, qemu and livbirt.

Shinobu

----- Original Message -----
From: "B, Naga Venkata" <naga.b@hpe.com>
To: "Shinobu Kinjo" <skinjo@redhat.com>
Cc: "Ceph Development" <ceph-devel@vger.kernel.org>
Sent: Thursday, August 27, 2015 9:47:16 PM
Subject: RE: Errors attaching RBD image to a running VM

NO, debian 3.14.44-1-amd64, QEMU emulator version 2.3.0, openstack kilo, libvirt 1.2.9, ceph version 0.80.7

I am to attach volume to nova instance if instance is in shutoff state. For that following steps I did.

I restarted libvirtd and libvirt-guests, after libvirt-guests restart vm went to shutoff state but nova is not updated instance state still it is in running state, now I did volume attach it went fine.
If I restart nova-compute instance state is updated with shutoff state, if I try to start instance the below error I am facing.
 
libvirtError: internal error: process exited while connecting to monitor: 2015-08-27T11:16:02.726265Z qemu-system-x86_64: -drive file=rbd:volumes/volume-ee7f6e3d-8269-4ffc-8d62-a881a0d7116f:id=cinder:key=AQCfEdxVUP13MBAAv4Oj7twFIxaF1BXyY1PkRQ==:auth_supported=cephx\;none:mon_host=192.168.245.2\:6789\;192.168.245.3\:6789\;192.168.245.4\:6789,if=none,id=drive-virtio-disk1,format=raw,serial=ee7f6e3d-8269-4ffc-8d62-a881a0d7116f,cache=none: Unknown protocol 'rbd'

Still this error "Errors attaching RBD image to a running VM" is there.

Thanks & Regards,
Venkat
-----Original Message-----
From: Shinobu Kinjo [mailto:skinjo@redhat.com] 
Sent: Thursday, August 27, 2015 6:00 PM
To: B, Naga Venkata
Cc: Ceph Development
Subject: Re: Errors attaching RBD image to a running VM

Hello,

My question from first look at is that, is it your environment?

> Currently running SL 6.3, OpenStack Folsom packages from EPEL, Libvirt 
> 0.9.10, QEMU 1.2.1 built from an epel-testing SRPM for RBD support, 
> and Ceph 0.54

If it's true, you have to upgrade SL and Ceph as much as you can.
Because, as you might notice, there have been a bunch of bug fixes.

And please attach logs related to Ceph, qemu and livbirt.

Shinobu

----- Original Message -----
From: "B, Naga Venkata" <naga.b@hpe.com>
To: "Shinobu Kinjo" <skinjo@redhat.com>
Cc: "Ceph Development" <ceph-devel@vger.kernel.org>
Sent: Thursday, August 27, 2015 7:12:36 PM
Subject: RE: Errors attaching RBD image to a running VM

Hi Shinobu,

Can I have quick reply on this, is this issue with libvirt version or ceph.

Thanks & Regards,
Venkat

-----Original Message-----
From: Shinobu Kinjo [mailto:skinjo@redhat.com]
Sent: Wednesday, August 26, 2015 5:54 PM
To: B, Naga Venkata
Cc: Ceph Development
Subject: Re: Errors attaching RBD image to a running VM

Hello Venkat,

Thank you!!

 Shinobu

----- Original Message -----
From: "B, Naga Venkata" <naga.b@hpe.com>
To: skinjo@redhat.com
Cc: "Ceph Development" <ceph-devel@vger.kernel.org>
Sent: Wednesday, August 26, 2015 7:24:17 PM
Subject: RE: Errors attaching RBD image to a running VM

Hi Shinobu,

This is from http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/11230.

Thanks & Regards,
Venkat


From: Shinobu [mailto:shinobu.kj@gmail.com]
Sent: Tuesday, August 25, 2015 5:35 PM
To: B, Naga Venkata
Cc: Ceph Development
Subject: Re: Errors attaching RBD image to a running VM

Where is original message?
Shinobu

On Tue, Aug 25, 2015 at 8:31 PM, venkat <naga.b@hp.com<mailto:naga.b@hp.com>> wrote:



same issue with libvirt version 1.2.9,
can somebody help me.

Thanks & Regards,
Venkat

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org<mailto:majordomo@vger.kernel.org>
More majordomo info at  http://vger.kernel.org/majordomo-info.html



--
Email:
 shinobu@linux.com<mailto:shinobu@linux.com>
 skinjo@redhat.com<mailto:skinjo@redhat.com>

 Life w/ Linux<http://i-shinobu.hatenablog.com/>