Re: Problem with attaching rbd device in qemu-kvm

[Josh Durgin <josh.durgin@dreamhost.com>]

On 12/13/2011 04:56 AM, Sławomir Skowron wrote:
> Finally, i manage the problem with rbd with kvm 1.0, and libvirt
> 0.9.8, or i think i manage :), but i get stuck with one thing after.
>
> 2011-12-13 12:13:31.173+0000: 21512: error :
> qemuMonitorJSONCheckError:318 : internal error unable to execute QEMU
> command 'device_add': Bus 'pci.0' does not support hotplugging
> 2011-12-13 12:13:31.173+0000: 21512: warning :
> qemuDomainAttachPciDiskDevice:244 : qemuMonitorAddDevice failed on
> file=rbd:rbd/testdysk:id=admin:key=AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog==:auth_supported=cephx
> none:mon_host=10.177.64.4\:6789,if=none,id=drive-virtio-disk2,format=raw
> (virtio-blk-pci,bus=pci.0,addr=0x7,drive=drive-virtio-disk2,id=virtio-disk2)
> 2011-12-13 12:13:31.173+0000: 21512: error :
> virSecurityDACRestoreSecurityFileLabel:143 : cannot resolve symlink
> rbd/testdysk: No such file or directory2011-12-13 12:13:31.173+0000:
> 21512: warning : qemuDomainAttachPciDiskDevice:287 : Unable to restore
> security label on rbd/testdysk
> and i can't find any info about that.
>
> my kvm.xml inject file:
>
> <disk type="network" device="disk">
>           <driver name="qemu" type="raw"/>
>           <auth username="admin">
>                   <secret type="ceph"
> uuid="0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f"/>
>           </auth>
>           <source protocol="rbd" name="rbd/testdysk">
>                   <host name="10.177.64.4" port="6789"/>
>           </source>
>           <target dev="sdc" bus="virtio"/>
> </disk>
>
> secret:
>
> cat /tmp/secret.xml
> <secret ephemeral='no' private='no'>
>       <uuid>0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f</uuid>
>       <usage type='ceph'>
>               <domain>rbd/admin</domain>
>               <name>admin</name>
>       </usage>
> </secret>
>
> virsh secret-define /tmp/secret.xml
> Secret 0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f created
>
> virsh secret-set-value 0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f
> AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog==
> Secret value set
>

Your xml and configuration are all fine, you're hitting the libvirt bug 
that I mentioned before. The fix isn't in any stable release yet, and 
0.9.8 was just released yesterday, so I guess you'll have to compile it 
yourself or disable the libvirt security driver.

> 2011/12/12 Josh Durgin<josh.durgin@dreamhost.com>:
>> On 12/09/2011 04:48 AM, Sławomir Skowron wrote:
>>>
>>> Sorry, for my lag, but i was sick.
>>>
>>> I handled problem with apparmor before and now it's not a problem,
>>> even when i send mail before, it was solved.
>>>
>>> Ok when i create image from qemu-img it's looks like that, and works
>>> perfect, (attachment with log of this opperation.)
>>>
>>> But when i use virsh, it's still a problem. Just like he don't know
>>> what user is going to be auth.
>>>
>>> I try, as client.admin (default i hope :)) only with id=admin in
>>> kvm.xml, or client=admin, and other combinations - like client.rbd
>>> etc., Still nothing.
>>
>>
>> admin is the default, and the correct option for client.admin is id=admin,
>> or id=foo for client.foo.
>>
>>
>>>
>>> I try, improve a keyring file, adding client rbd into keyring on
>>> machine with kvm hypervisor in /etc/ceph/keyring.bin
>>>
>>> [client.rbd]
>>>          key = AAAAAAAAAAAAAAAA
>>>          auid = 18446744073709551615
>>>          caps mds = "allow"
>>>          caps mon = "allow r"
>>>          caps osd = "allow rw pool=kvmtest"
>>>
>>> and on cluster site like above. I was added, a section [client.rbd]
>>> with keyring path, and it's still a problem.
>>>
>>
>> Did you run 'ceph auth add -i /etc/ceph/keyring.bin'?
>> Make sure 'ceph auth list' shows client.rbd with the right key.
>
> Overall i want to use admin id, and i think now it's correctly used in xml.
>
>>
>>
>>> 2011-12-09 13:29:11.519096 7fe616b59700 mon.0@0(leader) e1
>>> ms_verify_authorizer 10.177.32.66:0/58020608 client protocol 0
>>> root@s3-10-177-64-4:~# tail -n10000 /var/log/ceph/mon.0.log | grep
>>> 13:29:11.
>>> 2011-12-09 13:29:11.519096 7fe616b59700 mon.0@0(leader) e1
>>> ms_verify_authorizer 10.177.32.66:0/58020608 client protocol 0
>>> 2011-12-09 13:29:11.519624 7fe61b090700 -- 10.177.64.4:6789/0<==
>>> client.? 10.177.32.66:0/58020608 1 ==== auth(proto 0 21 bytes) v1 ====
>>> 47+0+0 (3310669357 0 0) 0x17f2e00 con 0x19d2c80
>>> 2011-12-09 13:29:11.519645 7fe61b090700 mon.0@0(leader) e1 have connection
>>> 2011-12-09 13:29:11.519655 7fe61b090700 mon.0@0(leader) e1 do not have
>>> session, making new one
>>> 2011-12-09 13:29:11.519668 7fe61b090700 mon.0@0(leader) e1 ms_dispatch
>>> new session MonSession: client.? 10.177.32.66:0/58020608 is open for
>>> client.? 10.177.32.66:0/58020608
>>> 2011-12-09 13:29:11.519674 7fe61b090700 mon.0@0(leader) e1 setting
>>> timeout on session
>>> 2011-12-09 13:29:11.519680 7fe61b090700 mon.0@0(leader) e1  caps
>>> 2011-12-09 13:29:11.519688 7fe61b090700 mon.0@0(leader).auth v353
>>> AuthMonitor::update_from_paxos()
>>> 2011-12-09 13:29:11.519697 7fe61b090700 mon.0@0(leader).auth v353
>>> preprocess_query auth(proto 0 21 bytes) v1 from client.?
>>> 10.177.32.66:0/58020608
>>> 2011-12-09 13:29:11.519703 7fe61b090700 mon.0@0(leader).auth v353
>>> prep_auth() blob_size=21
>>> 2011-12-09 13:29:11.519735 7fe61b090700 -- 10.177.64.4:6789/0 -->
>>> 10.177.32.66:0/58020608 -- auth_reply(proto 0 -95 Operation not
>>> supported) v1 -- ?+0 0x1246200 con 0x19d2c80
>>>
>>> from libvirt.log
>>>
>>> 23:20:56.689: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
>>> : cannot resolve symlink rbd:kvmtest/kvmtest1:name=admin: No such file
>>> or directory
>>> 23:20:56.882: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>>> Unable to restore security label on rbd:kvmtest/kvmtest1:name=admin
>>> 23:21:09.172: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>>> failed: open disk image file failed
>>> 23:21:09.172: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
>>> : cannot resolve symlink rbd:kvmtest/kvmtest1:name=admin: No such file
>>> or directory
>>> 23:21:09.364: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>>> Unable to restore security label on rbd:kvmtest/kvmtest1:name=admin
>>> 23:21:49.645: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>>> failed: open disk image file failed
>>> 23:21:49.645: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
>>> : cannot resolve symlink rbd:kvmtest/kvmtest1:id=admin: No such file
>>> or directory
>>> 23:21:49.853: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>>> Unable to restore security label on rbd:kvmtest/kvmtest1:id=admin
>>> 13:08:08.924: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>>> failed: open disk image file failed
>>> 13:08:08.924: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
>>> : cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or
>>> directory
>>> 13:08:09.117: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>>> Unable to restore security label on rbd:kvmtest/kvmtest1
>>> 13:09:45.402: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>>> failed: open disk image file failed
>>> 13:29:05.554: 1310: error : qemuMonitorTextAddDrive:2418 : operation
>>> failed: open disk image file failed
>>> 13:29:05.554: 1310: error : virSecurityDACRestoreSecurityFileLabel:143
>>> : cannot resolve symlink kvmtest/kvmtest1: No such file or directory
>>> 13:29:05.751: 1310: warning : qemuDomainAttachPciDiskDevice:250 :
>>> Unable to restore security label on kvmtest/kvmtest1
>>> 13:29:05.751: 1310: warning : virEventPollUpdateHandle:147 : Ignoring
>>> invalid update watch -1
>>
>>
>> This is libvirt mistakenly trying to treat the rbd disk as a file - fixed in
>> 20e1233c31e3150d259073c523077acdccb5d419 to libvirt. If you don't want to
>> upgrade libvirt, someone on IRC last week reported that installing
>> libapparmor1 fixed this problem for them, since libvirt no longer fell back
>> to it's own DAC implementation.
>
> hmm is this commited to any stable version now ?? i have installed
> libapparmor1,
>
> ri  libapparmor1                    2.7.0~beta1+bzr1774-1ubuntu2
>          changehat AppArmor library
>
> and i use newest stable libvirt 0.9.8.
>
>>
>>
>>> 2011/12/1 Josh Durgin<josh.durgin@dreamhost.com>:
>>>>
>>>> On 12/01/2011 11:37 AM, Sławomir Skowron wrote:
>>>>>
>>>>>
>>>>> I have some problems. Can you help me ??
>>>>>
>>>>> ceph cluster:
>>>>> ceph 0.38, oneiric, kernel 3.0.0 x86_64 - now only one machine.
>>>>>
>>>>> kvm hypervisor:
>>>>> kvm version 1.0-rc4 (0.15.92), libvirt 0.9.2, kernel 3.0.0, Ubuntu
>>>>> oneiric x86_64.
>>>>>
>>>>> I create image from qemu-img on machine with kvm VM's, and it works very
>>>>> well:
>>>>>
>>>>> qemu-img create -f rbd rbd:kvmtest/kvmtest1 10G
>>>>> Formatting 'rbd:kvmtest/kvmtest1', fmt=rbd size=10737418240
>>>>> cluster_size=0
>>>>>
>>>>> in ceph cluster machines:
>>>>>
>>>>> rados -p kvmtest ls
>>>>> kvmtest1.rbd
>>>>> rbd_directory
>>>>> rbd_info
>>>>>
>>>>> But when i try to attach new device to kvm VM like this:
>>>>>
>>>>> virsh attach-device one-10 /tmp/kvm.xml
>>>>>
>>>>> with kvm.xml like this:
>>>>>
>>>>> <disk type='network' device='disk'>
>>>>>          <driver name='qemu' type='raw' cache='writeback'/>
>>>>>          <source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
>>>>>                  <host name='10.177.64.4' port='6789'/>
>>>>>          </source>
>>>>>          <target dev='vdc' bus='virtio'/>
>>>>> </disk>
>>>>>
>>>>> output of virsh:
>>>>>
>>>>> error: Failed to attach device from /tmp/kvm.xml
>>>>> error: cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or
>>>>> directory
>>>>
>>>>
>>>>
>>>>   From the monitor output below, I'm assuming this is just a bad error
>>>> message
>>>> from libvirt.
>>>>
>>>>
>>>>> and mon.0 log in ceph cluster.
>>>>>
>>>>> 2011-12-01 20:18:17.142595 7fe02dd04700 mon.0@0(leader) e1
>>>>> ms_verify_authorizer 10.177.32.66:0/25020608 client protocol 0
>>>>> 2011-12-01 20:18:17.143041 7fe03223b700 -- 10.177.64.4:6789/0<==
>>>>> client.? 10.177.32.66:0/25020608 1 ==== auth(proto 0 21 bytes) v1 ====
>>>>> 47+0+0 (3310669357 0 0) 0x29f6a00 con 0x297adc0
>>>>> 2011-12-01 20:18:17.143095 7fe03223b700 mon.0@0(leader) e1 ms_dispatch
>>>>> new session MonSession: client.? 10.177.32.66:0/25020608 is open for
>>>>> client.? 10.177.32.66:0/25020608
>>>>> 2011-12-01 20:18:17.143125 7fe03223b700 mon.0@0(leader).auth v146
>>>>> preprocess_query auth(proto 0 21 bytes) v1 from client.?
>>>>> 10.177.32.66:0/25020608
>>>>> 2011-12-01 20:18:17.143168 7fe03223b700 -- 10.177.64.4:6789/0 -->
>>>>> 10.177.32.66:0/25020608 -- auth_reply(proto 0 -95 Operation not
>>>>> supported) v1 -- ?+0 0x2f30600 con 0x297adc0
>>>>
>>>>
>>>>
>>>> Here you can see qemu isn't authenticating correctly, probably because
>>>> the
>>>> rados user isn't being set by libvirt. With your libvirt version, you'll
>>>> need to change:
>>>>
>>>>
>>>>    <source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
>>>>
>>>> to
>>>>
>>>>    <source protocol='rbd' name='rbd:kvmtest/kvmtest1:id=foo'>
>>>>
>>>> to authenticate as client.foo. Any ceph config option can be set like
>>>> that,
>>>> separated by colons.
>>>>
>>>> If this still doesn't work, I'd suggest setting
>>>> debug_rbd=20:debug_monc=20:debug_auth=20:log_to_stderr=2 to get more info
>>>> in
>>>> the libvirt instance log (usually
>>>> /var/log/libvirt/qemu/instance_name.log).
>>>> There may be apparmour restrictions preventing qemu from reading the
>>>> keyring
>>>> or ceph config files.
>>>>
>>>>
>>>>> ceph.conf, and keyring.bin exists on server, and machine with qemu-kvm
>>>>> in same directory:
>>>>> ; global
>>>>> [global]
>>>>>          ; enable secure authentication
>>>>>
>>>>>          auth supported = cephx
>>>>>          keyring = /etc/ceph/keyring.bin
>>>>>
>>>>>          debug rgw = 1
>>>>>
>>>>>          rgw print continue = false
>>>>>          rgw socket path = /var/run/radosgw.sock
>>>>>
>>>>> ; monitors
>>>>> ;  You need at least one.  You need at least three if you want to
>>>>> ;  tolerate any node failures.  Always create an odd number.
>>>>> [mon]
>>>>>          mon data = /vol0/data/mon.$id
>>>>>
>>>>>          ; some minimal logging (just message traffic) to aid debugging
>>>>>
>>>>>          debug ms = 1     ; see message traffic
>>>>>          debug mon = 0   ; monitor
>>>>>          debug paxos = 0 ; monitor replication
>>>>>          debug auth = 0  ;
>>>>>
>>>>>          mon allowed clock drift = 2
>>>>>
>>>>> [mon.0]
>>>>>          host = 10-177-64-4
>>>>>          mon addr = 10.177.64.4:6789
>>>>>
>>>>> ; radosgw client list
>>>>> [client.radosgw.10-177-64-4]
>>>>>
>>>>>          host = 10-177-64-4
>>>>>          log file = /var/log/ceph/$name.log
>>>>>          debug rgw = 1
>>>>>          debug ms = 1
>>>>>
>>>>> ; osd
>>>>> ;  You need at least one.  Two if you want data to be replicated.
>>>>> ;  Define as many as you like.
>>>>> [osd]
>>>>>          ; This is where the btrfs volume will be mounted.
>>>>>
>>>>>          osd data = /vol0/data/osd.$id
>>>>>
>>>>>          ; Ideally, make this a separate disk or partition.  A few GB
>>>>>          ; is usually enough; more if you have fast disks.  You can use
>>>>>          ; a file under the osd data dir if need be
>>>>>          ; (e.g. /data/osd$id/journal), but it will be slower than a
>>>>>          ; separate disk or partition.
>>>>>
>>>>>          osd journal = /vol0/data/osd.$id/journal
>>>>>
>>>>>          ; If the OSD journal is a file, you need to specify the size.
>>>>> This is specified in MB.
>>>>>
>>>>>          osd journal size = 512
>>>>>
>>>>>          filestore journal writeahead = 1
>>>>>          osd heartbeat grace = 5
>>>>>
>>>>>          debug ms = 0         ; message traffic
>>>>>          debug osd = 0
>>>>>          debug filestore = 0 ; local object storage
>>>>>          debug journal = 0   ; local journaling
>>>>>          debug monc = 0
>>>>>          debug rados = 1
>>>>>
>>>>> [osd.0]
>>>>>          host = 10-177-64-4
>>>>>          osd data = /vol0/data/osd.0
>>>>>          keyring = /vol0/data/osd.0/keyring
>>>>>
>>>>> [osd.1]
>>>>>          host = 10-177-64-4
>>>>>          osd data = /vol0/data/osd.1
>>>>>          keyring = /vol0/data/osd.1/keyring
>>>>>
>>>>> [osd.2]
>>>>>          host = 10-177-64-4
>>>>>          osd data = /vol0/data/osd.2
>>>>>          keyring = /vol0/data/osd.2/keyring
>>>>>
>>>>> [osd.3]
>>>>>          host = 10-177-64-4
>>>>>          osd data = /vol0/data/osd.3
>>>>>          keyring = /vol0/data/osd.3/keyring
>>>>>
>>>>> [osd.4]
>>>>>          host = 10-177-64-4
>>>>>          osd data = /vol0/data/osd.4
>>>>>          keyring = /vol0/data/osd.4/keyring
>>>>>
>>>>> [osd.5]
>>>>>          host = 10-177-64-4
>>>>>          osd data = /vol0/data/osd.5
>>>>>          keyring = /vol0/data/osd.5/keyring
>>>>>
>>>>> [osd.6]
>>>>>          host = 10-177-64-4
>>>>>          osd data = /vol0/data/osd.6
>>>>>          keyring = /vol0/data/osd.6/keyring
>>>>>
>>>>> [osd.7]
>>>>>          host = 10-177-64-4
>>>>>          osd data = /vol0/data/osd.7
>>>>>          keyring = /vol0/data/osd.7/keyring
>>>>>
>>>>> [osd.8]
>>>>>          host = 10-177-64-4
>>>>>          osd data = /vol0/data/osd.8
>>>>>          keyring = /vol0/data/osd.8/keyring
>>>>>
>>>>> [osd.9]
>>>>>          host = 10-177-64-4
>>>>>          osd data = /vol0/data/osd.9
>>>>>          keyring = /vol0/data/osd.9/keyring
>>>>>
>>>>> [osd.10]
>>>>>          host = 10-177-64-4
>>>>>          osd data = /vol0/data/osd.10
>>>>>          keyring = /vol0/data/osd.10/keyring
>>>>>
>>>>> [osd.11]
>>>>>          host = 10-177-64-4
>>>>>          osd data = /vol0/data/osd.11
>>>>>          keyring = /vol0/data/osd.11/keyring
>>>>>
>>>>> [osd.12]
>>>>>          host = 10-177-64-4
>>>>>          osd data = /vol0/data/osd.12
>>>>>          keyring = /vol0/data/osd.12/keyring
>>>>>
>>>>> [osd.13]
>>>>>          host = 10-177-64-4
>>>>>          osd data = /vol0/data/osd.13
>>>>>          keyring = /vol0/data/osd.13/keyring
>>>>>
>>>>> [osd.14]
>>>>>          host = 10-177-64-4
>>>>>          osd data = /vol0/data/osd.14
>>>>>          keyring = /vol0/data/osd.14/keyring
>>>>>
>>>>> [osd.15]
>>>>>          host = 10-177-64-4
>>>>>          osd data = /vol0/data/osd.15
>>>>>          keyring = /vol0/data/osd.15/keyring
>>>>>
>>>>> [osd.16]
>>>>>          host = 10-177-64-4
>>>>>          osd data = /vol0/data/osd.16
>>>>>          keyring = /vol0/data/osd.16/keyring
>>>>>
>>>>> [osd.17]
>>>>>          host = 10-177-64-4
>>>>>          osd data = /vol0/data/osd.17
>>>>>          keyring = /vol0/data/osd.17/keyring
>>>>>
>>>>> [osd.18]
>>>>>          host = 10-177-64-4
>>>>>          osd data = /vol0/data/osd.18
>>>>>          keyring = /vol0/data/osd.18/keyring
>>>>>
>>>>> [osd.19]
>>>>>          host = 10-177-64-4
>>>>>          osd data = /vol0/data/osd.19
>>>>>          keyring = /vol0/data/osd.19/keyring
>>>>>
>>>>> [osd.20]
>>>>>          host = 10-177-64-4
>>>>>          osd data = /vol0/data/osd.20
>>>>>          keyring = /vol0/data/osd.20/keyring
>>>>>
>>>>> [osd.21]
>>>>>          host = 10-177-64-4
>>>>>          osd data = /vol0/data/osd.21
>>>>>          keyring = /vol0/data/osd.21/keyring
>>>>>
>>>>> [osd.22]
>>>>>          host = 10-177-64-4
>>>>>          osd data = /vol0/data/osd.22
>>>>>          keyring = /vol0/data/osd.22/keyring
>>>>>
>>>>> [osd.23]
>>>>>          host = 10-177-64-4
>>>>>          osd data = /vol0/data/osd.23
>>>>>          keyring = /vol0/data/osd.23/keyring
>>>>>
>>>>> [osd.24]
>>>>>          host = 10-177-64-4
>>>>>          osd data = /vol0/data/osd.24
>>>>>          keyring = /vol0/data/osd.24/keyring
>>>>>
>>>>> [osd.25]
>>>>>          host = 10-177-64-4
>>>>>          osd data = /vol0/data/osd.25
>>>>>          keyring = /vol0/data/osd.25/keyring
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>
>
>
>

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html