Re: Problem with attaching rbd device in qemu-kvm

[Josh Durgin <josh.durgin@dreamhost.com>]

On 12/09/2011 04:48 AM, Sławomir Skowron wrote:
> Sorry, for my lag, but i was sick.
>
> I handled problem with apparmor before and now it's not a problem,
> even when i send mail before, it was solved.
>
> Ok when i create image from qemu-img it's looks like that, and works
> perfect, (attachment with log of this opperation.)
>
> But when i use virsh, it's still a problem. Just like he don't know
> what user is going to be auth.
>
> I try, as client.admin (default i hope :)) only with id=admin in
> kvm.xml, or client=admin, and other combinations - like client.rbd
> etc., Still nothing.

admin is the default, and the correct option for client.admin is 
id=admin, or id=foo for client.foo.

>
> I try, improve a keyring file, adding client rbd into keyring on
> machine with kvm hypervisor in /etc/ceph/keyring.bin
>
> [client.rbd]
>          key = AAAAAAAAAAAAAAAA
>          auid = 18446744073709551615
>          caps mds = "allow"
>          caps mon = "allow r"
>          caps osd = "allow rw pool=kvmtest"
>
> and on cluster site like above. I was added, a section [client.rbd]
> with keyring path, and it's still a problem.
>

Did you run 'ceph auth add -i /etc/ceph/keyring.bin'?
Make sure 'ceph auth list' shows client.rbd with the right key.

> 2011-12-09 13:29:11.519096 7fe616b59700 mon.0@0(leader) e1
> ms_verify_authorizer 10.177.32.66:0/58020608 client protocol 0
> root@s3-10-177-64-4:~# tail -n10000 /var/log/ceph/mon.0.log | grep 13:29:11.
> 2011-12-09 13:29:11.519096 7fe616b59700 mon.0@0(leader) e1
> ms_verify_authorizer 10.177.32.66:0/58020608 client protocol 0
> 2011-12-09 13:29:11.519624 7fe61b090700 -- 10.177.64.4:6789/0<==
> client.? 10.177.32.66:0/58020608 1 ==== auth(proto 0 21 bytes) v1 ====
> 47+0+0 (3310669357 0 0) 0x17f2e00 con 0x19d2c80
> 2011-12-09 13:29:11.519645 7fe61b090700 mon.0@0(leader) e1 have connection
> 2011-12-09 13:29:11.519655 7fe61b090700 mon.0@0(leader) e1 do not have
> session, making new one
> 2011-12-09 13:29:11.519668 7fe61b090700 mon.0@0(leader) e1 ms_dispatch
> new session MonSession: client.? 10.177.32.66:0/58020608 is open for
> client.? 10.177.32.66:0/58020608
> 2011-12-09 13:29:11.519674 7fe61b090700 mon.0@0(leader) e1 setting
> timeout on session
> 2011-12-09 13:29:11.519680 7fe61b090700 mon.0@0(leader) e1  caps
> 2011-12-09 13:29:11.519688 7fe61b090700 mon.0@0(leader).auth v353
> AuthMonitor::update_from_paxos()
> 2011-12-09 13:29:11.519697 7fe61b090700 mon.0@0(leader).auth v353
> preprocess_query auth(proto 0 21 bytes) v1 from client.?
> 10.177.32.66:0/58020608
> 2011-12-09 13:29:11.519703 7fe61b090700 mon.0@0(leader).auth v353
> prep_auth() blob_size=21
> 2011-12-09 13:29:11.519735 7fe61b090700 -- 10.177.64.4:6789/0 -->
> 10.177.32.66:0/58020608 -- auth_reply(proto 0 -95 Operation not
> supported) v1 -- ?+0 0x1246200 con 0x19d2c80
>
> from libvirt.log
>
> 23:20:56.689: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
> : cannot resolve symlink rbd:kvmtest/kvmtest1:name=admin: No such file
> or directory
> 23:20:56.882: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
> Unable to restore security label on rbd:kvmtest/kvmtest1:name=admin
> 23:21:09.172: 1309: error : qemuMonitorTextAddDrive:2418 : operation
> failed: open disk image file failed
> 23:21:09.172: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
> : cannot resolve symlink rbd:kvmtest/kvmtest1:name=admin: No such file
> or directory
> 23:21:09.364: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
> Unable to restore security label on rbd:kvmtest/kvmtest1:name=admin
> 23:21:49.645: 1309: error : qemuMonitorTextAddDrive:2418 : operation
> failed: open disk image file failed
> 23:21:49.645: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
> : cannot resolve symlink rbd:kvmtest/kvmtest1:id=admin: No such file
> or directory
> 23:21:49.853: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
> Unable to restore security label on rbd:kvmtest/kvmtest1:id=admin
> 13:08:08.924: 1309: error : qemuMonitorTextAddDrive:2418 : operation
> failed: open disk image file failed
> 13:08:08.924: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
> : cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or
> directory
> 13:08:09.117: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
> Unable to restore security label on rbd:kvmtest/kvmtest1
> 13:09:45.402: 1309: error : qemuMonitorTextAddDrive:2418 : operation
> failed: open disk image file failed
> 13:29:05.554: 1310: error : qemuMonitorTextAddDrive:2418 : operation
> failed: open disk image file failed
> 13:29:05.554: 1310: error : virSecurityDACRestoreSecurityFileLabel:143
> : cannot resolve symlink kvmtest/kvmtest1: No such file or directory
> 13:29:05.751: 1310: warning : qemuDomainAttachPciDiskDevice:250 :
> Unable to restore security label on kvmtest/kvmtest1
> 13:29:05.751: 1310: warning : virEventPollUpdateHandle:147 : Ignoring
> invalid update watch -1

This is libvirt mistakenly trying to treat the rbd disk as a file - 
fixed in 20e1233c31e3150d259073c523077acdccb5d419 to libvirt. If you 
don't want to upgrade libvirt, someone on IRC last week reported that 
installing libapparmor1 fixed this problem for them, since libvirt no 
longer fell back to it's own DAC implementation.

> 2011/12/1 Josh Durgin<josh.durgin@dreamhost.com>:
>> On 12/01/2011 11:37 AM, Sławomir Skowron wrote:
>>>
>>> I have some problems. Can you help me ??
>>>
>>> ceph cluster:
>>> ceph 0.38, oneiric, kernel 3.0.0 x86_64 - now only one machine.
>>>
>>> kvm hypervisor:
>>> kvm version 1.0-rc4 (0.15.92), libvirt 0.9.2, kernel 3.0.0, Ubuntu
>>> oneiric x86_64.
>>>
>>> I create image from qemu-img on machine with kvm VM's, and it works very
>>> well:
>>>
>>> qemu-img create -f rbd rbd:kvmtest/kvmtest1 10G
>>> Formatting 'rbd:kvmtest/kvmtest1', fmt=rbd size=10737418240 cluster_size=0
>>>
>>> in ceph cluster machines:
>>>
>>> rados -p kvmtest ls
>>> kvmtest1.rbd
>>> rbd_directory
>>> rbd_info
>>>
>>> But when i try to attach new device to kvm VM like this:
>>>
>>> virsh attach-device one-10 /tmp/kvm.xml
>>>
>>> with kvm.xml like this:
>>>
>>> <disk type='network' device='disk'>
>>>          <driver name='qemu' type='raw' cache='writeback'/>
>>>          <source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
>>>                  <host name='10.177.64.4' port='6789'/>
>>>          </source>
>>>          <target dev='vdc' bus='virtio'/>
>>> </disk>
>>>
>>> output of virsh:
>>>
>>> error: Failed to attach device from /tmp/kvm.xml
>>> error: cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or
>>> directory
>>
>>
>>  From the monitor output below, I'm assuming this is just a bad error message
>> from libvirt.
>>
>>
>>> and mon.0 log in ceph cluster.
>>>
>>> 2011-12-01 20:18:17.142595 7fe02dd04700 mon.0@0(leader) e1
>>> ms_verify_authorizer 10.177.32.66:0/25020608 client protocol 0
>>> 2011-12-01 20:18:17.143041 7fe03223b700 -- 10.177.64.4:6789/0<==
>>> client.? 10.177.32.66:0/25020608 1 ==== auth(proto 0 21 bytes) v1 ====
>>> 47+0+0 (3310669357 0 0) 0x29f6a00 con 0x297adc0
>>> 2011-12-01 20:18:17.143095 7fe03223b700 mon.0@0(leader) e1 ms_dispatch
>>> new session MonSession: client.? 10.177.32.66:0/25020608 is open for
>>> client.? 10.177.32.66:0/25020608
>>> 2011-12-01 20:18:17.143125 7fe03223b700 mon.0@0(leader).auth v146
>>> preprocess_query auth(proto 0 21 bytes) v1 from client.?
>>> 10.177.32.66:0/25020608
>>> 2011-12-01 20:18:17.143168 7fe03223b700 -- 10.177.64.4:6789/0 -->
>>> 10.177.32.66:0/25020608 -- auth_reply(proto 0 -95 Operation not
>>> supported) v1 -- ?+0 0x2f30600 con 0x297adc0
>>
>>
>> Here you can see qemu isn't authenticating correctly, probably because the
>> rados user isn't being set by libvirt. With your libvirt version, you'll
>> need to change:
>>
>>
>>    <source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
>>
>> to
>>
>>    <source protocol='rbd' name='rbd:kvmtest/kvmtest1:id=foo'>
>>
>> to authenticate as client.foo. Any ceph config option can be set like that,
>> separated by colons.
>>
>> If this still doesn't work, I'd suggest setting
>> debug_rbd=20:debug_monc=20:debug_auth=20:log_to_stderr=2 to get more info in
>> the libvirt instance log (usually /var/log/libvirt/qemu/instance_name.log).
>> There may be apparmour restrictions preventing qemu from reading the keyring
>> or ceph config files.
>>
>>
>>> ceph.conf, and keyring.bin exists on server, and machine with qemu-kvm
>>> in same directory:
>>> ; global
>>> [global]
>>>          ; enable secure authentication
>>>
>>>          auth supported = cephx
>>>          keyring = /etc/ceph/keyring.bin
>>>
>>>          debug rgw = 1
>>>
>>>          rgw print continue = false
>>>          rgw socket path = /var/run/radosgw.sock
>>>
>>> ; monitors
>>> ;  You need at least one.  You need at least three if you want to
>>> ;  tolerate any node failures.  Always create an odd number.
>>> [mon]
>>>          mon data = /vol0/data/mon.$id
>>>
>>>          ; some minimal logging (just message traffic) to aid debugging
>>>
>>>          debug ms = 1     ; see message traffic
>>>          debug mon = 0   ; monitor
>>>          debug paxos = 0 ; monitor replication
>>>          debug auth = 0  ;
>>>
>>>          mon allowed clock drift = 2
>>>
>>> [mon.0]
>>>          host = 10-177-64-4
>>>          mon addr = 10.177.64.4:6789
>>>
>>> ; radosgw client list
>>> [client.radosgw.10-177-64-4]
>>>
>>>          host = 10-177-64-4
>>>          log file = /var/log/ceph/$name.log
>>>          debug rgw = 1
>>>          debug ms = 1
>>>
>>> ; osd
>>> ;  You need at least one.  Two if you want data to be replicated.
>>> ;  Define as many as you like.
>>> [osd]
>>>          ; This is where the btrfs volume will be mounted.
>>>
>>>          osd data = /vol0/data/osd.$id
>>>
>>>          ; Ideally, make this a separate disk or partition.  A few GB
>>>          ; is usually enough; more if you have fast disks.  You can use
>>>          ; a file under the osd data dir if need be
>>>          ; (e.g. /data/osd$id/journal), but it will be slower than a
>>>          ; separate disk or partition.
>>>
>>>          osd journal = /vol0/data/osd.$id/journal
>>>
>>>          ; If the OSD journal is a file, you need to specify the size.
>>> This is specified in MB.
>>>
>>>          osd journal size = 512
>>>
>>>          filestore journal writeahead = 1
>>>          osd heartbeat grace = 5
>>>
>>>          debug ms = 0         ; message traffic
>>>          debug osd = 0
>>>          debug filestore = 0 ; local object storage
>>>          debug journal = 0   ; local journaling
>>>          debug monc = 0
>>>          debug rados = 1
>>>
>>> [osd.0]
>>>          host = 10-177-64-4
>>>          osd data = /vol0/data/osd.0
>>>          keyring = /vol0/data/osd.0/keyring
>>>
>>> [osd.1]
>>>          host = 10-177-64-4
>>>          osd data = /vol0/data/osd.1
>>>          keyring = /vol0/data/osd.1/keyring
>>>
>>> [osd.2]
>>>          host = 10-177-64-4
>>>          osd data = /vol0/data/osd.2
>>>          keyring = /vol0/data/osd.2/keyring
>>>
>>> [osd.3]
>>>          host = 10-177-64-4
>>>          osd data = /vol0/data/osd.3
>>>          keyring = /vol0/data/osd.3/keyring
>>>
>>> [osd.4]
>>>          host = 10-177-64-4
>>>          osd data = /vol0/data/osd.4
>>>          keyring = /vol0/data/osd.4/keyring
>>>
>>> [osd.5]
>>>          host = 10-177-64-4
>>>          osd data = /vol0/data/osd.5
>>>          keyring = /vol0/data/osd.5/keyring
>>>
>>> [osd.6]
>>>          host = 10-177-64-4
>>>          osd data = /vol0/data/osd.6
>>>          keyring = /vol0/data/osd.6/keyring
>>>
>>> [osd.7]
>>>          host = 10-177-64-4
>>>          osd data = /vol0/data/osd.7
>>>          keyring = /vol0/data/osd.7/keyring
>>>
>>> [osd.8]
>>>          host = 10-177-64-4
>>>          osd data = /vol0/data/osd.8
>>>          keyring = /vol0/data/osd.8/keyring
>>>
>>> [osd.9]
>>>          host = 10-177-64-4
>>>          osd data = /vol0/data/osd.9
>>>          keyring = /vol0/data/osd.9/keyring
>>>
>>> [osd.10]
>>>          host = 10-177-64-4
>>>          osd data = /vol0/data/osd.10
>>>          keyring = /vol0/data/osd.10/keyring
>>>
>>> [osd.11]
>>>          host = 10-177-64-4
>>>          osd data = /vol0/data/osd.11
>>>          keyring = /vol0/data/osd.11/keyring
>>>
>>> [osd.12]
>>>          host = 10-177-64-4
>>>          osd data = /vol0/data/osd.12
>>>          keyring = /vol0/data/osd.12/keyring
>>>
>>> [osd.13]
>>>          host = 10-177-64-4
>>>          osd data = /vol0/data/osd.13
>>>          keyring = /vol0/data/osd.13/keyring
>>>
>>> [osd.14]
>>>          host = 10-177-64-4
>>>          osd data = /vol0/data/osd.14
>>>          keyring = /vol0/data/osd.14/keyring
>>>
>>> [osd.15]
>>>          host = 10-177-64-4
>>>          osd data = /vol0/data/osd.15
>>>          keyring = /vol0/data/osd.15/keyring
>>>
>>> [osd.16]
>>>          host = 10-177-64-4
>>>          osd data = /vol0/data/osd.16
>>>          keyring = /vol0/data/osd.16/keyring
>>>
>>> [osd.17]
>>>          host = 10-177-64-4
>>>          osd data = /vol0/data/osd.17
>>>          keyring = /vol0/data/osd.17/keyring
>>>
>>> [osd.18]
>>>          host = 10-177-64-4
>>>          osd data = /vol0/data/osd.18
>>>          keyring = /vol0/data/osd.18/keyring
>>>
>>> [osd.19]
>>>          host = 10-177-64-4
>>>          osd data = /vol0/data/osd.19
>>>          keyring = /vol0/data/osd.19/keyring
>>>
>>> [osd.20]
>>>          host = 10-177-64-4
>>>          osd data = /vol0/data/osd.20
>>>          keyring = /vol0/data/osd.20/keyring
>>>
>>> [osd.21]
>>>          host = 10-177-64-4
>>>          osd data = /vol0/data/osd.21
>>>          keyring = /vol0/data/osd.21/keyring
>>>
>>> [osd.22]
>>>          host = 10-177-64-4
>>>          osd data = /vol0/data/osd.22
>>>          keyring = /vol0/data/osd.22/keyring
>>>
>>> [osd.23]
>>>          host = 10-177-64-4
>>>          osd data = /vol0/data/osd.23
>>>          keyring = /vol0/data/osd.23/keyring
>>>
>>> [osd.24]
>>>          host = 10-177-64-4
>>>          osd data = /vol0/data/osd.24
>>>          keyring = /vol0/data/osd.24/keyring
>>>
>>> [osd.25]
>>>          host = 10-177-64-4
>>>          osd data = /vol0/data/osd.25
>>>          keyring = /vol0/data/osd.25/keyring
>>>
>>>
>>
>
>
>

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html