Re: about attaching rbd volume from instance on KVM

[Josh Durgin <josh.durgin@dreamhost.com>]

On 02/15/2012 07:01 PM, Masuko Tomoya wrote:
> Hi Josh,
>
> Thank you for your reply !
>
>> libvirt 0.9.9 fixes this (the applied patch is http://www.redhat.com/archives/libvir-list/2011-December/msg00506.html if you're curious).
> Finally I applied the patch to libvirt 0.9.8 and
> I fixed it.
>
> But I have another issue.
> When I tried attach rbd volume to VM on KVM, error occured.
>
> --
> root@compute04:~# virsh attach-device instance-00000027 testvolume4.xml
> error: Failed to attach device from testvolume4.xml
> error: operation failed: open disk image file failed
> --
>
> Could you advice to me ?
>
> *testvolume4.xml is below.
> <disk type='network' device='disk'>
>    <driver name='qemu' type='raw'/>
>    <source protocol='rbd' name='rbd/testvolume3'>
>      <host name='10.2.100.56' port='6789'/>
>    </source>
>    <target dev='vdm' bus='virtio'/>
> </disk>
>
> * I use this qemu-kvm with built rbd option on.
> root@compute04:/usr/bin# /usr/bin/qemu-system-x86_64 -version
> QEMU emulator version 1.0 (qemu-kvm-1.0), Copyright (c) 2003-2008
> Fabrice Bellard
>
> *libvirtd.log is below
> 2012-02-15 05:30:24.669+0000: 14034: debug :
> remoteDispatchDomainAttachDeviceHelper:297 : server=0x1dad7f0
> client=0x1e78fd0 msg=0x7fedd0003490 rerr=0x7feddeb1fc70 args=0x1e6f270
> ret=0x1e6f210
> 2012-02-15 05:30:24.669+0000: 14032: debug : virEventPollRunOnce:619 :
> EVENT_POLL_RUN: nhandles=9 imeout=5000
> 2012-02-15 05:30:24.669+0000: 14034: debug :
> virDomainAttachDevice:8542 : dom=0x1e6f230, (VM:
> name=instance-00000027, uuid=51846781-0d41-44a4-9da7-902c7c394f53),
> xml=<disk type='network' device='disk'>
>    <driver name='qemu' type='raw'/>
>    <source protocol='rbd' name='rbd/testvolume3'>
>      <host name='10.2.100.56' port='6789'/>
>    </source>
>    <target dev='vdm' bus='virtio'/>
> </disk>
>

Your xml looks fine, but libvirt has a problem:

> buf={"execute":"human-monitor-command","arguments":{"command-line":"drive_add
> dummy file=rbd:rbd/testvolume3:mon_host=10.2.100.56\\\\:6789,if=none,id=drive-virtio-disk12,format=raw"},"id":"libvirt-8"}
>   len=198 ret=198 errno=11

It looks like "10.2.100.56:6789" has been escaped twice. This makes qemu 
interpret the configuration as mon_host = 10.2.100.56\\, and then, since 
the colon is unescaped, 6789 is read as a separate option:

> 2012-02-15 05:30:24.674+0000: 14032: debug : qemuMonitorIOProcess:313
> : QEMU_MONITOR_IO_PROCESS: mon=0x1dcdf70 buf={"return": "conf option
> 6789 has no value\r\nerror setting config options\r\ncould not open
> disk image rbd:rbd/testvolume3:mon_host=10.2.100.56\\\\:6789: Invalid
> argument\r\n", "id": "libvirt-8"}
>   len=197
>

I'll look into the right way to solve this next week, but if you want a 
workaround, you can apply this patch to libvirt:

diff --git a/src/qemu/qemu_monitor_text.c b/src/qemu/qemu_monitor_text.c
index 2c68be8..3530ff4 100644
--- a/src/qemu/qemu_monitor_text.c
+++ b/src/qemu/qemu_monitor_text.c
@@ -2508,20 +2508,13 @@ int qemuMonitorTextAttachDrive(qemuMonitorPtr mon,
      char *cmd = NULL;
      char *reply = NULL;
      int ret = -1;
-    char *safe_str;
      int tryOldSyntax = 0;

-    safe_str = qemuMonitorEscapeArg(drivestr);
-    if (!safe_str) {
-        virReportOOMError();
-        return -1;
-    }
-
  try_command:
      if (virAsprintf(&cmd, "drive_add %s%.2x:%.2x:%.2x %s",
                      (tryOldSyntax ? "" : "pci_addr="),
                      controllerAddr->domain, controllerAddr->bus,
-                    controllerAddr->slot, safe_str) < 0) {
+                    controllerAddr->slot, drivestr) < 0) {
          virReportOOMError();
          goto cleanup;
      }
@@ -2555,7 +2548,6 @@ try_command:
  cleanup:
      VIR_FREE(cmd);
      VIR_FREE(reply);
-    VIR_FREE(safe_str);
      return ret;
  }

>
> Waiting for reply,
>
> Tomoya.
>
> 2012/2/9 Josh Durgin<josh.durgin@dreamhost.com>:
>> On 02/08/2012 03:42 PM, Masuko Tomoya wrote:
>>>
>>> Hi Christian,
>>>
>>> Thank you for reply !
>>>
>>> Could I ask a version of libvirt which is your patch applied ?
>>>
>>> I'm trying it with libvirt0.9.8-2ubuntu9, but there is same error.
>>> (I'm using blow package:
>>> https://launchpad.net/ubuntu/precise/+source/libvirt/0.9.8-2ubuntu9)
>>>
>>> root@compute04:~# dpkg -l | grep libvirt-bin
>>> ii  libvirt-bin                            0.9.8-2ubuntu9
>>>              programs for the libvirt library
>>>
>>> /var/log/libvirtd/libvirt.log
>>> 2012-02-08 23:33:53.800+0000: 4663: error :
>>> qemuMonitorTextAddDrive:2828 : operation failed: open disk image file
>>> failed
>>> 2012-02-08 23:33:53.800+0000: 4663: error :
>>> virSecurityDACRestoreSecurityFileLabel:143 : cannot resolve symlink
>>> rbd/testvolume: No such file or directory
>>> 2012-02-08 23:33:54.212+0000: 4663: warning :
>>> qemuDomainAttachPciDiskDevice:287 : Unable to restore security label
>>> on rbd/testvolume
>>
>>
>> libvirt 0.9.9 fixes this (the applied patch is
>> http://www.redhat.com/archives/libvir-list/2011-December/msg00506.html if
>> you're curious).
>>
>>
>>>
>>> 2012/2/6 Christian Brunner<christian@brunner-muc.de>:
>>>>
>>>> Libvirt is trying to set security lables even for network shares. This
>>>> will not work. I think this is fixed in newer libvirt versions. For
>>>> older versions you can try this patch:
>>>>
>>>> http://www.redhat.com/archives/libvir-list/2011-May/msg01446.html
>>>>
>>>> Regards,
>>>> Christian
>>>>
>>>> 2012/2/4 Masuko Tomoya<tomoya.masuko@gmail.com>:
>>>>>
>>>>> Hi, all.
>>>>>
>>>>> I'm trying to attach rbd volume from instance on KVM.
>>>>> But I have problem.
>>>>> Could you help me ?
>>>>>
>>>>> ---
>>>>> I tried to attach rbd volume on ceph01 to instance on compute1 with
>>>>> virsh command.
>>>>>
>>>>> root@compute1:~# virsh attach-device test-ub16 /root/testvolume.xml
>>>>> error: Failed to attach device from /root/testvolume.xml
>>>>> error: cannot resolve symlink rbd/testvolume: No such file or directory
>>>>>
>>>>> /var/log/messages
>>>>> Feb  3 20:14:48 compute1 libvirtd: 20:14:48.717: 3234: error :
>>>>> qemuMonitorTextAddDevice:2417 : operation failed: adding
>>>>>
>>>>> virtio-blk-pci,bus=pci.0,addr=0x9,drive=drive-virtio-disk4,id=virtio-disk4
>>>>> device failed: Device needs media, but drive is empty#015#012Device
>>>>> 'virtio-blk-pci' could not be initialized#015#012
>>>>> Feb  3 20:14:48 compute1 libvirtd: 20:14:48.717: 3234: warning :
>>>>> qemuDomainAttachPciDiskDevice:188 : qemuMonitorAddDevice failed on
>>>>> file=rbd:rbd/testvolume,if=none,id=drive-virtio-disk4,format=raw
>>>>>
>>>>> (virtio-blk-pci,bus=pci.0,addr=0x9,drive=drive-virtio-disk4,id=virtio-disk4)
>>>>> Feb  3 20:14:48 compute1 libvirtd: 20:14:48.717: 3234: error :
>>>>> virSecurityDACRestoreSecurityFileLabel:143 : cannot resolve symlink
>>>>> rbd/testvolume: No such file or directory
>>>>> Feb  3 20:14:48 compute1 libvirtd: 20:14:48.717: 3234: warning :
>>>>> qemuDomainAttachPciDiskDevice:229 : Unable to restore security label
>>>>> on rbd/testvolume
>>>>>
>>>>> there is no log in /var/log/ceph/mon.0.log of host ceph01.
>>>>> ---
>>>>>
>>>>>
>>>>> My environment is below.
>>>>> *There are two servers. All server are ubuntu 10.10 x86_64.
>>>>> *ceph01: single server configured ceph.(version: 0.41-1maverick)
>>>>> *compute1: kvm hypervisor
>>>>>   -librados2 and librbd1 packages are installed.
>>>>>   (version: 0.41-1maverick)
>>>>>   -qemu-kvm is 0.14.0-rc1. I built qemu with rbd enable.
>>>>>   the output of run 'qemu-img' show 'rbd' at supported formats field.
>>>>>   (I built qemu reffering this page.
>>>>>   http://ceph.newdream.net/wiki/QEMU-RBD)
>>>>>   -apparmor is disable.
>>>>>   -libvirt is 0.8.8
>>>>>
>>>>> ====
>>>>>   -there is ceph.conf on compute1.
>>>>> root@compute1:~# ls -l /etc/ceph/
>>>>> total 20
>>>>> -rw-r--r-- 1 root root 508 2012-02-03 14:38 ceph.conf
>>>>> -rw------- 1 root root  63 2012-02-03 17:04 keyring.admin
>>>>> -rw------- 1 root root  63 2012-02-03 14:38 keyring.bin
>>>>> -rw------- 1 root root  56 2012-02-03 14:38 keyring.mds.0
>>>>> -rw------- 1 root root  56 2012-02-03 14:38 keyring.osd.0
>>>>>
>>>>> =====
>>>>>   -contents of ceph.conf is below.
>>>>> root@compute1:~# cat /etc/ceph/ceph.conf
>>>>> [global]
>>>>>        auth supported = cephx
>>>>>        keyring = /etc/ceph/keyring.bin
>>>>> [mon]
>>>>>        mon data = /data/data/mon$id
>>>>>        debug ms = 1
>>>>> [mon.0]
>>>>>        host = ceph01
>>>>>        mon addr = 10.68.119.191:6789
>>>>> [mds]
>>>>>        keyring = /etc/ceph/keyring.$name
>>>>> [mds.0]
>>>>>        host = ceph01
>>>>> [osd]
>>>>>        keyring = /etc/ceph/keyring.$name
>>>>>        osd data = /data/osd$id
>>>>>        osd journal = /data/osd$id/journal
>>>>>        osd journal size = 512
>>>>>        osd class tmp = /var/lib/ceph/tmp
>>>>>        debug osd = 20
>>>>>        debug ms = 1
>>>>>        debug filestore = 20
>>>>> [osd.0]
>>>>>        host = ceph01
>>>>>        btrfs devs = /dev/sdb1
>>>>>
>>>>> ===
>>>>> *conten of keyring.admin is below
>>>>> root@compute1:~# cat /etc/ceph/keyring.admin
>>>>> [client.admin]
>>>>>        key = AQDFeCxPyBlNIRAAxS1DcRHpMXRpcjY/GNMwYg==
>>>>>
>>>>>
>>>>> ===
>>>>> *output of run 'ceph auth list'
>>>>> root@ceph01:/etc/ceph# ceph auth list
>>>>> 2012-02-03 20:34:59.507451 mon<- [auth,list]
>>>>> 2012-02-03 20:34:59.508785 mon.0 ->    'installed auth entries:
>>>>> mon.
>>>>>        key: AQDFeCxPiK04IxAAslDBNkrOGKWxcbCh2iysqg==
>>>>> mds.0
>>>>>        key: AQDFeCxPsJ+LGhAAJ3/rmkAtGXSv/eHh0yXgww==
>>>>>        caps: [mds] allow
>>>>>        caps: [mon] allow rwx
>>>>>        caps: [osd] allow *
>>>>> osd.0
>>>>>        key: AQDFeCxPoEK+ExAAecD7+tWgpIRoZx2AT7Jwbg==
>>>>>        caps: [mon] allow rwx
>>>>>        caps: [osd] allow *
>>>>> client.admin
>>>>>        key: AQDFeCxPyBlNIRAAxS1DcRHpMXRpcjY/GNMwYg==
>>>>>        caps: [mds] allow
>>>>>        caps: [mon] allow *
>>>>>        caps: [osd] allow *
>>>>> ' (0)
>>>>>
>>>>> ====
>>>>> *xml file is below.
>>>>> root@compute1:~# cat /root/testvolume.xml
>>>>> <disk type='network' device='disk'>
>>>>>   <driver name='qemu' type='raw'/>
>>>>>   <source protocol='rbd' name='rbd/testvolume'>
>>>>>    <host name='10.68.119.191' port='6789'/>
>>>>>   </source>
>>>>>   <target dev='vde' bus='virtio'/>
>>>>> </disk>
>>>>>
>>>>> ====
>>>>> *testvolume is on rados pools.
>>>>> root@compute1:~# qemu-img info rbd:rbd/testvolume
>>>>> image: rbd:rbd/testvolume
>>>>> file format: raw
>>>>> virtual size: 1.0G (1073741824 bytes)
>>>>> disk size: unavailable
>>>>>
>>>>>
>>>>> Waiting for reply,
>>>>>
>>>>> Tomoya.