Re: Problem with attaching rbd device in qemu-kvm

[Josh Durgin <josh.durgin@dreamhost.com>]

On 12/01/2011 11:37 AM, Sławomir Skowron wrote:
> I have some problems. Can you help me ??
>
> ceph cluster:
> ceph 0.38, oneiric, kernel 3.0.0 x86_64 - now only one machine.
>
> kvm hypervisor:
> kvm version 1.0-rc4 (0.15.92), libvirt 0.9.2, kernel 3.0.0, Ubuntu
> oneiric x86_64.
>
> I create image from qemu-img on machine with kvm VM's, and it works very well:
>
> qemu-img create -f rbd rbd:kvmtest/kvmtest1 10G
> Formatting 'rbd:kvmtest/kvmtest1', fmt=rbd size=10737418240 cluster_size=0
>
> in ceph cluster machines:
>
> rados -p kvmtest ls
> kvmtest1.rbd
> rbd_directory
> rbd_info
>
> But when i try to attach new device to kvm VM like this:
>
> virsh attach-device one-10 /tmp/kvm.xml
>
> with kvm.xml like this:
>
> <disk type='network' device='disk'>
>          <driver name='qemu' type='raw' cache='writeback'/>
>          <source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
>                  <host name='10.177.64.4' port='6789'/>
>          </source>
>          <target dev='vdc' bus='virtio'/>
> </disk>
>
> output of virsh:
>
> error: Failed to attach device from /tmp/kvm.xml
> error: cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or directory

 From the monitor output below, I'm assuming this is just a bad error 
message from libvirt.

> and mon.0 log in ceph cluster.
>
> 2011-12-01 20:18:17.142595 7fe02dd04700 mon.0@0(leader) e1
> ms_verify_authorizer 10.177.32.66:0/25020608 client protocol 0
> 2011-12-01 20:18:17.143041 7fe03223b700 -- 10.177.64.4:6789/0<==
> client.? 10.177.32.66:0/25020608 1 ==== auth(proto 0 21 bytes) v1 ====
> 47+0+0 (3310669357 0 0) 0x29f6a00 con 0x297adc0
> 2011-12-01 20:18:17.143095 7fe03223b700 mon.0@0(leader) e1 ms_dispatch
> new session MonSession: client.? 10.177.32.66:0/25020608 is open for
> client.? 10.177.32.66:0/25020608
> 2011-12-01 20:18:17.143125 7fe03223b700 mon.0@0(leader).auth v146
> preprocess_query auth(proto 0 21 bytes) v1 from client.?
> 10.177.32.66:0/25020608
> 2011-12-01 20:18:17.143168 7fe03223b700 -- 10.177.64.4:6789/0 -->
> 10.177.32.66:0/25020608 -- auth_reply(proto 0 -95 Operation not
> supported) v1 -- ?+0 0x2f30600 con 0x297adc0

Here you can see qemu isn't authenticating correctly, probably because 
the rados user isn't being set by libvirt. With your libvirt version, 
you'll need to change:

    <source protocol='rbd' name='rbd:kvmtest/kvmtest1'>

to

    <source protocol='rbd' name='rbd:kvmtest/kvmtest1:id=foo'>

to authenticate as client.foo. Any ceph config option can be set like 
that, separated by colons.

If this still doesn't work, I'd suggest setting 
debug_rbd=20:debug_monc=20:debug_auth=20:log_to_stderr=2 to get more 
info in the libvirt instance log (usually 
/var/log/libvirt/qemu/instance_name.log). There may be apparmour 
restrictions preventing qemu from reading the keyring or ceph config files.

> ceph.conf, and keyring.bin exists on server, and machine with qemu-kvm
> in same directory:
> ; global
> [global]
>          ; enable secure authentication
>
>          auth supported = cephx
>          keyring = /etc/ceph/keyring.bin
>
>          debug rgw = 1
>
>          rgw print continue = false
>          rgw socket path = /var/run/radosgw.sock
>
> ; monitors
> ;  You need at least one.  You need at least three if you want to
> ;  tolerate any node failures.  Always create an odd number.
> [mon]
>          mon data = /vol0/data/mon.$id
>
>          ; some minimal logging (just message traffic) to aid debugging
>
>          debug ms = 1     ; see message traffic
>          debug mon = 0   ; monitor
>          debug paxos = 0 ; monitor replication
>          debug auth = 0  ;
>
>          mon allowed clock drift = 2
>
> [mon.0]
>          host = 10-177-64-4
>          mon addr = 10.177.64.4:6789
>
> ; radosgw client list
> [client.radosgw.10-177-64-4]
>
>          host = 10-177-64-4
>          log file = /var/log/ceph/$name.log
>          debug rgw = 1
>          debug ms = 1
>
> ; osd
> ;  You need at least one.  Two if you want data to be replicated.
> ;  Define as many as you like.
> [osd]
>          ; This is where the btrfs volume will be mounted.
>
>          osd data = /vol0/data/osd.$id
>
>          ; Ideally, make this a separate disk or partition.  A few GB
>          ; is usually enough; more if you have fast disks.  You can use
>          ; a file under the osd data dir if need be
>          ; (e.g. /data/osd$id/journal), but it will be slower than a
>          ; separate disk or partition.
>
>          osd journal = /vol0/data/osd.$id/journal
>
>          ; If the OSD journal is a file, you need to specify the size.
> This is specified in MB.
>
>          osd journal size = 512
>
>          filestore journal writeahead = 1
>          osd heartbeat grace = 5
>
>          debug ms = 0         ; message traffic
>          debug osd = 0
>          debug filestore = 0 ; local object storage
>          debug journal = 0   ; local journaling
>          debug monc = 0
>          debug rados = 1
>
> [osd.0]
>          host = 10-177-64-4
>          osd data = /vol0/data/osd.0
>          keyring = /vol0/data/osd.0/keyring
>
> [osd.1]
>          host = 10-177-64-4
>          osd data = /vol0/data/osd.1
>          keyring = /vol0/data/osd.1/keyring
>
> [osd.2]
>          host = 10-177-64-4
>          osd data = /vol0/data/osd.2
>          keyring = /vol0/data/osd.2/keyring
>
> [osd.3]
>          host = 10-177-64-4
>          osd data = /vol0/data/osd.3
>          keyring = /vol0/data/osd.3/keyring
>
> [osd.4]
>          host = 10-177-64-4
>          osd data = /vol0/data/osd.4
>          keyring = /vol0/data/osd.4/keyring
>
> [osd.5]
>          host = 10-177-64-4
>          osd data = /vol0/data/osd.5
>          keyring = /vol0/data/osd.5/keyring
>
> [osd.6]
>          host = 10-177-64-4
>          osd data = /vol0/data/osd.6
>          keyring = /vol0/data/osd.6/keyring
>
> [osd.7]
>          host = 10-177-64-4
>          osd data = /vol0/data/osd.7
>          keyring = /vol0/data/osd.7/keyring
>
> [osd.8]
>          host = 10-177-64-4
>          osd data = /vol0/data/osd.8
>          keyring = /vol0/data/osd.8/keyring
>
> [osd.9]
>          host = 10-177-64-4
>          osd data = /vol0/data/osd.9
>          keyring = /vol0/data/osd.9/keyring
>
> [osd.10]
>          host = 10-177-64-4
>          osd data = /vol0/data/osd.10
>          keyring = /vol0/data/osd.10/keyring
>
> [osd.11]
>          host = 10-177-64-4
>          osd data = /vol0/data/osd.11
>          keyring = /vol0/data/osd.11/keyring
>
> [osd.12]
>          host = 10-177-64-4
>          osd data = /vol0/data/osd.12
>          keyring = /vol0/data/osd.12/keyring
>
> [osd.13]
>          host = 10-177-64-4
>          osd data = /vol0/data/osd.13
>          keyring = /vol0/data/osd.13/keyring
>
> [osd.14]
>          host = 10-177-64-4
>          osd data = /vol0/data/osd.14
>          keyring = /vol0/data/osd.14/keyring
>
> [osd.15]
>          host = 10-177-64-4
>          osd data = /vol0/data/osd.15
>          keyring = /vol0/data/osd.15/keyring
>
> [osd.16]
>          host = 10-177-64-4
>          osd data = /vol0/data/osd.16
>          keyring = /vol0/data/osd.16/keyring
>
> [osd.17]
>          host = 10-177-64-4
>          osd data = /vol0/data/osd.17
>          keyring = /vol0/data/osd.17/keyring
>
> [osd.18]
>          host = 10-177-64-4
>          osd data = /vol0/data/osd.18
>          keyring = /vol0/data/osd.18/keyring
>
> [osd.19]
>          host = 10-177-64-4
>          osd data = /vol0/data/osd.19
>          keyring = /vol0/data/osd.19/keyring
>
> [osd.20]
>          host = 10-177-64-4
>          osd data = /vol0/data/osd.20
>          keyring = /vol0/data/osd.20/keyring
>
> [osd.21]
>          host = 10-177-64-4
>          osd data = /vol0/data/osd.21
>          keyring = /vol0/data/osd.21/keyring
>
> [osd.22]
>          host = 10-177-64-4
>          osd data = /vol0/data/osd.22
>          keyring = /vol0/data/osd.22/keyring
>
> [osd.23]
>          host = 10-177-64-4
>          osd data = /vol0/data/osd.23
>          keyring = /vol0/data/osd.23/keyring
>
> [osd.24]
>          host = 10-177-64-4
>          osd data = /vol0/data/osd.24
>          keyring = /vol0/data/osd.24/keyring
>
> [osd.25]
>          host = 10-177-64-4
>          osd data = /vol0/data/osd.25
>          keyring = /vol0/data/osd.25/keyring
>
>

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html