* Problem with attaching rbd device in qemu-kvm
@ 2011-12-01 19:37 Sławomir Skowron
2011-12-01 20:42 ` Josh Durgin
0 siblings, 1 reply; 10+ messages in thread
From: Sławomir Skowron @ 2011-12-01 19:37 UTC (permalink / raw)
To: ceph-devel
I have some problems. Can you help me ??
ceph cluster:
ceph 0.38, oneiric, kernel 3.0.0 x86_64 - now only one machine.
kvm hypervisor:
kvm version 1.0-rc4 (0.15.92), libvirt 0.9.2, kernel 3.0.0, Ubuntu
oneiric x86_64.
I create image from qemu-img on machine with kvm VM's, and it works very well:
qemu-img create -f rbd rbd:kvmtest/kvmtest1 10G
Formatting 'rbd:kvmtest/kvmtest1', fmt=rbd size=10737418240 cluster_size=0
in ceph cluster machines:
rados -p kvmtest ls
kvmtest1.rbd
rbd_directory
rbd_info
But when i try to attach new device to kvm VM like this:
virsh attach-device one-10 /tmp/kvm.xml
with kvm.xml like this:
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='writeback'/>
<source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
<host name='10.177.64.4' port='6789'/>
</source>
<target dev='vdc' bus='virtio'/>
</disk>
output of virsh:
error: Failed to attach device from /tmp/kvm.xml
error: cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or directory
and mon.0 log in ceph cluster.
2011-12-01 20:18:17.142595 7fe02dd04700 mon.0@0(leader) e1
ms_verify_authorizer 10.177.32.66:0/25020608 client protocol 0
2011-12-01 20:18:17.143041 7fe03223b700 -- 10.177.64.4:6789/0 <==
client.? 10.177.32.66:0/25020608 1 ==== auth(proto 0 21 bytes) v1 ====
47+0+0 (3310669357 0 0) 0x29f6a00 con 0x297adc0
2011-12-01 20:18:17.143095 7fe03223b700 mon.0@0(leader) e1 ms_dispatch
new session MonSession: client.? 10.177.32.66:0/25020608 is open for
client.? 10.177.32.66:0/25020608
2011-12-01 20:18:17.143125 7fe03223b700 mon.0@0(leader).auth v146
preprocess_query auth(proto 0 21 bytes) v1 from client.?
10.177.32.66:0/25020608
2011-12-01 20:18:17.143168 7fe03223b700 -- 10.177.64.4:6789/0 -->
10.177.32.66:0/25020608 -- auth_reply(proto 0 -95 Operation not
supported) v1 -- ?+0 0x2f30600 con 0x297adc0
ceph.conf, and keyring.bin exists on server, and machine with qemu-kvm
in same directory:
; global
[global]
; enable secure authentication
auth supported = cephx
keyring = /etc/ceph/keyring.bin
debug rgw = 1
rgw print continue = false
rgw socket path = /var/run/radosgw.sock
; monitors
; You need at least one. You need at least three if you want to
; tolerate any node failures. Always create an odd number.
[mon]
mon data = /vol0/data/mon.$id
; some minimal logging (just message traffic) to aid debugging
debug ms = 1 ; see message traffic
debug mon = 0 ; monitor
debug paxos = 0 ; monitor replication
debug auth = 0 ;
mon allowed clock drift = 2
[mon.0]
host = 10-177-64-4
mon addr = 10.177.64.4:6789
; radosgw client list
[client.radosgw.10-177-64-4]
host = 10-177-64-4
log file = /var/log/ceph/$name.log
debug rgw = 1
debug ms = 1
; osd
; You need at least one. Two if you want data to be replicated.
; Define as many as you like.
[osd]
; This is where the btrfs volume will be mounted.
osd data = /vol0/data/osd.$id
; Ideally, make this a separate disk or partition. A few GB
; is usually enough; more if you have fast disks. You can use
; a file under the osd data dir if need be
; (e.g. /data/osd$id/journal), but it will be slower than a
; separate disk or partition.
osd journal = /vol0/data/osd.$id/journal
; If the OSD journal is a file, you need to specify the size.
This is specified in MB.
osd journal size = 512
filestore journal writeahead = 1
osd heartbeat grace = 5
debug ms = 0 ; message traffic
debug osd = 0
debug filestore = 0 ; local object storage
debug journal = 0 ; local journaling
debug monc = 0
debug rados = 1
[osd.0]
host = 10-177-64-4
osd data = /vol0/data/osd.0
keyring = /vol0/data/osd.0/keyring
[osd.1]
host = 10-177-64-4
osd data = /vol0/data/osd.1
keyring = /vol0/data/osd.1/keyring
[osd.2]
host = 10-177-64-4
osd data = /vol0/data/osd.2
keyring = /vol0/data/osd.2/keyring
[osd.3]
host = 10-177-64-4
osd data = /vol0/data/osd.3
keyring = /vol0/data/osd.3/keyring
[osd.4]
host = 10-177-64-4
osd data = /vol0/data/osd.4
keyring = /vol0/data/osd.4/keyring
[osd.5]
host = 10-177-64-4
osd data = /vol0/data/osd.5
keyring = /vol0/data/osd.5/keyring
[osd.6]
host = 10-177-64-4
osd data = /vol0/data/osd.6
keyring = /vol0/data/osd.6/keyring
[osd.7]
host = 10-177-64-4
osd data = /vol0/data/osd.7
keyring = /vol0/data/osd.7/keyring
[osd.8]
host = 10-177-64-4
osd data = /vol0/data/osd.8
keyring = /vol0/data/osd.8/keyring
[osd.9]
host = 10-177-64-4
osd data = /vol0/data/osd.9
keyring = /vol0/data/osd.9/keyring
[osd.10]
host = 10-177-64-4
osd data = /vol0/data/osd.10
keyring = /vol0/data/osd.10/keyring
[osd.11]
host = 10-177-64-4
osd data = /vol0/data/osd.11
keyring = /vol0/data/osd.11/keyring
[osd.12]
host = 10-177-64-4
osd data = /vol0/data/osd.12
keyring = /vol0/data/osd.12/keyring
[osd.13]
host = 10-177-64-4
osd data = /vol0/data/osd.13
keyring = /vol0/data/osd.13/keyring
[osd.14]
host = 10-177-64-4
osd data = /vol0/data/osd.14
keyring = /vol0/data/osd.14/keyring
[osd.15]
host = 10-177-64-4
osd data = /vol0/data/osd.15
keyring = /vol0/data/osd.15/keyring
[osd.16]
host = 10-177-64-4
osd data = /vol0/data/osd.16
keyring = /vol0/data/osd.16/keyring
[osd.17]
host = 10-177-64-4
osd data = /vol0/data/osd.17
keyring = /vol0/data/osd.17/keyring
[osd.18]
host = 10-177-64-4
osd data = /vol0/data/osd.18
keyring = /vol0/data/osd.18/keyring
[osd.19]
host = 10-177-64-4
osd data = /vol0/data/osd.19
keyring = /vol0/data/osd.19/keyring
[osd.20]
host = 10-177-64-4
osd data = /vol0/data/osd.20
keyring = /vol0/data/osd.20/keyring
[osd.21]
host = 10-177-64-4
osd data = /vol0/data/osd.21
keyring = /vol0/data/osd.21/keyring
[osd.22]
host = 10-177-64-4
osd data = /vol0/data/osd.22
keyring = /vol0/data/osd.22/keyring
[osd.23]
host = 10-177-64-4
osd data = /vol0/data/osd.23
keyring = /vol0/data/osd.23/keyring
[osd.24]
host = 10-177-64-4
osd data = /vol0/data/osd.24
keyring = /vol0/data/osd.24/keyring
[osd.25]
host = 10-177-64-4
osd data = /vol0/data/osd.25
keyring = /vol0/data/osd.25/keyring
--
-----
Pozdrawiam
Sławek "sZiBis" Skowron
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Problem with attaching rbd device in qemu-kvm
2011-12-01 19:37 Problem with attaching rbd device in qemu-kvm Sławomir Skowron
@ 2011-12-01 20:42 ` Josh Durgin
2011-12-09 12:48 ` Sławomir Skowron
0 siblings, 1 reply; 10+ messages in thread
From: Josh Durgin @ 2011-12-01 20:42 UTC (permalink / raw)
To: Sławomir Skowron; +Cc: ceph-devel
On 12/01/2011 11:37 AM, Sławomir Skowron wrote:
> I have some problems. Can you help me ??
>
> ceph cluster:
> ceph 0.38, oneiric, kernel 3.0.0 x86_64 - now only one machine.
>
> kvm hypervisor:
> kvm version 1.0-rc4 (0.15.92), libvirt 0.9.2, kernel 3.0.0, Ubuntu
> oneiric x86_64.
>
> I create image from qemu-img on machine with kvm VM's, and it works very well:
>
> qemu-img create -f rbd rbd:kvmtest/kvmtest1 10G
> Formatting 'rbd:kvmtest/kvmtest1', fmt=rbd size=10737418240 cluster_size=0
>
> in ceph cluster machines:
>
> rados -p kvmtest ls
> kvmtest1.rbd
> rbd_directory
> rbd_info
>
> But when i try to attach new device to kvm VM like this:
>
> virsh attach-device one-10 /tmp/kvm.xml
>
> with kvm.xml like this:
>
> <disk type='network' device='disk'>
> <driver name='qemu' type='raw' cache='writeback'/>
> <source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
> <host name='10.177.64.4' port='6789'/>
> </source>
> <target dev='vdc' bus='virtio'/>
> </disk>
>
> output of virsh:
>
> error: Failed to attach device from /tmp/kvm.xml
> error: cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or directory
From the monitor output below, I'm assuming this is just a bad error
message from libvirt.
> and mon.0 log in ceph cluster.
>
> 2011-12-01 20:18:17.142595 7fe02dd04700 mon.0@0(leader) e1
> ms_verify_authorizer 10.177.32.66:0/25020608 client protocol 0
> 2011-12-01 20:18:17.143041 7fe03223b700 -- 10.177.64.4:6789/0<==
> client.? 10.177.32.66:0/25020608 1 ==== auth(proto 0 21 bytes) v1 ====
> 47+0+0 (3310669357 0 0) 0x29f6a00 con 0x297adc0
> 2011-12-01 20:18:17.143095 7fe03223b700 mon.0@0(leader) e1 ms_dispatch
> new session MonSession: client.? 10.177.32.66:0/25020608 is open for
> client.? 10.177.32.66:0/25020608
> 2011-12-01 20:18:17.143125 7fe03223b700 mon.0@0(leader).auth v146
> preprocess_query auth(proto 0 21 bytes) v1 from client.?
> 10.177.32.66:0/25020608
> 2011-12-01 20:18:17.143168 7fe03223b700 -- 10.177.64.4:6789/0 -->
> 10.177.32.66:0/25020608 -- auth_reply(proto 0 -95 Operation not
> supported) v1 -- ?+0 0x2f30600 con 0x297adc0
Here you can see qemu isn't authenticating correctly, probably because
the rados user isn't being set by libvirt. With your libvirt version,
you'll need to change:
<source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
to
<source protocol='rbd' name='rbd:kvmtest/kvmtest1:id=foo'>
to authenticate as client.foo. Any ceph config option can be set like
that, separated by colons.
If this still doesn't work, I'd suggest setting
debug_rbd=20:debug_monc=20:debug_auth=20:log_to_stderr=2 to get more
info in the libvirt instance log (usually
/var/log/libvirt/qemu/instance_name.log). There may be apparmour
restrictions preventing qemu from reading the keyring or ceph config files.
> ceph.conf, and keyring.bin exists on server, and machine with qemu-kvm
> in same directory:
> ; global
> [global]
> ; enable secure authentication
>
> auth supported = cephx
> keyring = /etc/ceph/keyring.bin
>
> debug rgw = 1
>
> rgw print continue = false
> rgw socket path = /var/run/radosgw.sock
>
> ; monitors
> ; You need at least one. You need at least three if you want to
> ; tolerate any node failures. Always create an odd number.
> [mon]
> mon data = /vol0/data/mon.$id
>
> ; some minimal logging (just message traffic) to aid debugging
>
> debug ms = 1 ; see message traffic
> debug mon = 0 ; monitor
> debug paxos = 0 ; monitor replication
> debug auth = 0 ;
>
> mon allowed clock drift = 2
>
> [mon.0]
> host = 10-177-64-4
> mon addr = 10.177.64.4:6789
>
> ; radosgw client list
> [client.radosgw.10-177-64-4]
>
> host = 10-177-64-4
> log file = /var/log/ceph/$name.log
> debug rgw = 1
> debug ms = 1
>
> ; osd
> ; You need at least one. Two if you want data to be replicated.
> ; Define as many as you like.
> [osd]
> ; This is where the btrfs volume will be mounted.
>
> osd data = /vol0/data/osd.$id
>
> ; Ideally, make this a separate disk or partition. A few GB
> ; is usually enough; more if you have fast disks. You can use
> ; a file under the osd data dir if need be
> ; (e.g. /data/osd$id/journal), but it will be slower than a
> ; separate disk or partition.
>
> osd journal = /vol0/data/osd.$id/journal
>
> ; If the OSD journal is a file, you need to specify the size.
> This is specified in MB.
>
> osd journal size = 512
>
> filestore journal writeahead = 1
> osd heartbeat grace = 5
>
> debug ms = 0 ; message traffic
> debug osd = 0
> debug filestore = 0 ; local object storage
> debug journal = 0 ; local journaling
> debug monc = 0
> debug rados = 1
>
> [osd.0]
> host = 10-177-64-4
> osd data = /vol0/data/osd.0
> keyring = /vol0/data/osd.0/keyring
>
> [osd.1]
> host = 10-177-64-4
> osd data = /vol0/data/osd.1
> keyring = /vol0/data/osd.1/keyring
>
> [osd.2]
> host = 10-177-64-4
> osd data = /vol0/data/osd.2
> keyring = /vol0/data/osd.2/keyring
>
> [osd.3]
> host = 10-177-64-4
> osd data = /vol0/data/osd.3
> keyring = /vol0/data/osd.3/keyring
>
> [osd.4]
> host = 10-177-64-4
> osd data = /vol0/data/osd.4
> keyring = /vol0/data/osd.4/keyring
>
> [osd.5]
> host = 10-177-64-4
> osd data = /vol0/data/osd.5
> keyring = /vol0/data/osd.5/keyring
>
> [osd.6]
> host = 10-177-64-4
> osd data = /vol0/data/osd.6
> keyring = /vol0/data/osd.6/keyring
>
> [osd.7]
> host = 10-177-64-4
> osd data = /vol0/data/osd.7
> keyring = /vol0/data/osd.7/keyring
>
> [osd.8]
> host = 10-177-64-4
> osd data = /vol0/data/osd.8
> keyring = /vol0/data/osd.8/keyring
>
> [osd.9]
> host = 10-177-64-4
> osd data = /vol0/data/osd.9
> keyring = /vol0/data/osd.9/keyring
>
> [osd.10]
> host = 10-177-64-4
> osd data = /vol0/data/osd.10
> keyring = /vol0/data/osd.10/keyring
>
> [osd.11]
> host = 10-177-64-4
> osd data = /vol0/data/osd.11
> keyring = /vol0/data/osd.11/keyring
>
> [osd.12]
> host = 10-177-64-4
> osd data = /vol0/data/osd.12
> keyring = /vol0/data/osd.12/keyring
>
> [osd.13]
> host = 10-177-64-4
> osd data = /vol0/data/osd.13
> keyring = /vol0/data/osd.13/keyring
>
> [osd.14]
> host = 10-177-64-4
> osd data = /vol0/data/osd.14
> keyring = /vol0/data/osd.14/keyring
>
> [osd.15]
> host = 10-177-64-4
> osd data = /vol0/data/osd.15
> keyring = /vol0/data/osd.15/keyring
>
> [osd.16]
> host = 10-177-64-4
> osd data = /vol0/data/osd.16
> keyring = /vol0/data/osd.16/keyring
>
> [osd.17]
> host = 10-177-64-4
> osd data = /vol0/data/osd.17
> keyring = /vol0/data/osd.17/keyring
>
> [osd.18]
> host = 10-177-64-4
> osd data = /vol0/data/osd.18
> keyring = /vol0/data/osd.18/keyring
>
> [osd.19]
> host = 10-177-64-4
> osd data = /vol0/data/osd.19
> keyring = /vol0/data/osd.19/keyring
>
> [osd.20]
> host = 10-177-64-4
> osd data = /vol0/data/osd.20
> keyring = /vol0/data/osd.20/keyring
>
> [osd.21]
> host = 10-177-64-4
> osd data = /vol0/data/osd.21
> keyring = /vol0/data/osd.21/keyring
>
> [osd.22]
> host = 10-177-64-4
> osd data = /vol0/data/osd.22
> keyring = /vol0/data/osd.22/keyring
>
> [osd.23]
> host = 10-177-64-4
> osd data = /vol0/data/osd.23
> keyring = /vol0/data/osd.23/keyring
>
> [osd.24]
> host = 10-177-64-4
> osd data = /vol0/data/osd.24
> keyring = /vol0/data/osd.24/keyring
>
> [osd.25]
> host = 10-177-64-4
> osd data = /vol0/data/osd.25
> keyring = /vol0/data/osd.25/keyring
>
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Problem with attaching rbd device in qemu-kvm
2011-12-01 20:42 ` Josh Durgin
@ 2011-12-09 12:48 ` Sławomir Skowron
2011-12-12 18:53 ` Josh Durgin
0 siblings, 1 reply; 10+ messages in thread
From: Sławomir Skowron @ 2011-12-09 12:48 UTC (permalink / raw)
To: Josh Durgin; +Cc: ceph-devel
[-- Attachment #1: Type: text/plain, Size: 13488 bytes --]
Sorry, for my lag, but i was sick.
I handled problem with apparmor before and now it's not a problem,
even when i send mail before, it was solved.
Ok when i create image from qemu-img it's looks like that, and works
perfect, (attachment with log of this opperation.)
But when i use virsh, it's still a problem. Just like he don't know
what user is going to be auth.
I try, as client.admin (default i hope :)) only with id=admin in
kvm.xml, or client=admin, and other combinations - like client.rbd
etc., Still nothing.
I try, improve a keyring file, adding client rbd into keyring on
machine with kvm hypervisor in /etc/ceph/keyring.bin
[client.rbd]
key = AAAAAAAAAAAAAAAA
auid = 18446744073709551615
caps mds = "allow"
caps mon = "allow r"
caps osd = "allow rw pool=kvmtest"
and on cluster site like above. I was added, a section [client.rbd]
with keyring path, and it's still a problem.
2011-12-09 13:29:11.519096 7fe616b59700 mon.0@0(leader) e1
ms_verify_authorizer 10.177.32.66:0/58020608 client protocol 0
root@s3-10-177-64-4:~# tail -n10000 /var/log/ceph/mon.0.log | grep 13:29:11.
2011-12-09 13:29:11.519096 7fe616b59700 mon.0@0(leader) e1
ms_verify_authorizer 10.177.32.66:0/58020608 client protocol 0
2011-12-09 13:29:11.519624 7fe61b090700 -- 10.177.64.4:6789/0 <==
client.? 10.177.32.66:0/58020608 1 ==== auth(proto 0 21 bytes) v1 ====
47+0+0 (3310669357 0 0) 0x17f2e00 con 0x19d2c80
2011-12-09 13:29:11.519645 7fe61b090700 mon.0@0(leader) e1 have connection
2011-12-09 13:29:11.519655 7fe61b090700 mon.0@0(leader) e1 do not have
session, making new one
2011-12-09 13:29:11.519668 7fe61b090700 mon.0@0(leader) e1 ms_dispatch
new session MonSession: client.? 10.177.32.66:0/58020608 is open for
client.? 10.177.32.66:0/58020608
2011-12-09 13:29:11.519674 7fe61b090700 mon.0@0(leader) e1 setting
timeout on session
2011-12-09 13:29:11.519680 7fe61b090700 mon.0@0(leader) e1 caps
2011-12-09 13:29:11.519688 7fe61b090700 mon.0@0(leader).auth v353
AuthMonitor::update_from_paxos()
2011-12-09 13:29:11.519697 7fe61b090700 mon.0@0(leader).auth v353
preprocess_query auth(proto 0 21 bytes) v1 from client.?
10.177.32.66:0/58020608
2011-12-09 13:29:11.519703 7fe61b090700 mon.0@0(leader).auth v353
prep_auth() blob_size=21
2011-12-09 13:29:11.519735 7fe61b090700 -- 10.177.64.4:6789/0 -->
10.177.32.66:0/58020608 -- auth_reply(proto 0 -95 Operation not
supported) v1 -- ?+0 0x1246200 con 0x19d2c80
from libvirt.log
23:20:56.689: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
: cannot resolve symlink rbd:kvmtest/kvmtest1:name=admin: No such file
or directory
23:20:56.882: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
Unable to restore security label on rbd:kvmtest/kvmtest1:name=admin
23:21:09.172: 1309: error : qemuMonitorTextAddDrive:2418 : operation
failed: open disk image file failed
23:21:09.172: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
: cannot resolve symlink rbd:kvmtest/kvmtest1:name=admin: No such file
or directory
23:21:09.364: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
Unable to restore security label on rbd:kvmtest/kvmtest1:name=admin
23:21:49.645: 1309: error : qemuMonitorTextAddDrive:2418 : operation
failed: open disk image file failed
23:21:49.645: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
: cannot resolve symlink rbd:kvmtest/kvmtest1:id=admin: No such file
or directory
23:21:49.853: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
Unable to restore security label on rbd:kvmtest/kvmtest1:id=admin
13:08:08.924: 1309: error : qemuMonitorTextAddDrive:2418 : operation
failed: open disk image file failed
13:08:08.924: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
: cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or
directory
13:08:09.117: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
Unable to restore security label on rbd:kvmtest/kvmtest1
13:09:45.402: 1309: error : qemuMonitorTextAddDrive:2418 : operation
failed: open disk image file failed
13:29:05.554: 1310: error : qemuMonitorTextAddDrive:2418 : operation
failed: open disk image file failed
13:29:05.554: 1310: error : virSecurityDACRestoreSecurityFileLabel:143
: cannot resolve symlink kvmtest/kvmtest1: No such file or directory
13:29:05.751: 1310: warning : qemuDomainAttachPciDiskDevice:250 :
Unable to restore security label on kvmtest/kvmtest1
13:29:05.751: 1310: warning : virEventPollUpdateHandle:147 : Ignoring
invalid update watch -1
2011/12/1 Josh Durgin <josh.durgin@dreamhost.com>:
> On 12/01/2011 11:37 AM, Sławomir Skowron wrote:
>>
>> I have some problems. Can you help me ??
>>
>> ceph cluster:
>> ceph 0.38, oneiric, kernel 3.0.0 x86_64 - now only one machine.
>>
>> kvm hypervisor:
>> kvm version 1.0-rc4 (0.15.92), libvirt 0.9.2, kernel 3.0.0, Ubuntu
>> oneiric x86_64.
>>
>> I create image from qemu-img on machine with kvm VM's, and it works very
>> well:
>>
>> qemu-img create -f rbd rbd:kvmtest/kvmtest1 10G
>> Formatting 'rbd:kvmtest/kvmtest1', fmt=rbd size=10737418240 cluster_size=0
>>
>> in ceph cluster machines:
>>
>> rados -p kvmtest ls
>> kvmtest1.rbd
>> rbd_directory
>> rbd_info
>>
>> But when i try to attach new device to kvm VM like this:
>>
>> virsh attach-device one-10 /tmp/kvm.xml
>>
>> with kvm.xml like this:
>>
>> <disk type='network' device='disk'>
>> <driver name='qemu' type='raw' cache='writeback'/>
>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
>> <host name='10.177.64.4' port='6789'/>
>> </source>
>> <target dev='vdc' bus='virtio'/>
>> </disk>
>>
>> output of virsh:
>>
>> error: Failed to attach device from /tmp/kvm.xml
>> error: cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or
>> directory
>
>
> From the monitor output below, I'm assuming this is just a bad error message
> from libvirt.
>
>
>> and mon.0 log in ceph cluster.
>>
>> 2011-12-01 20:18:17.142595 7fe02dd04700 mon.0@0(leader) e1
>> ms_verify_authorizer 10.177.32.66:0/25020608 client protocol 0
>> 2011-12-01 20:18:17.143041 7fe03223b700 -- 10.177.64.4:6789/0<==
>> client.? 10.177.32.66:0/25020608 1 ==== auth(proto 0 21 bytes) v1 ====
>> 47+0+0 (3310669357 0 0) 0x29f6a00 con 0x297adc0
>> 2011-12-01 20:18:17.143095 7fe03223b700 mon.0@0(leader) e1 ms_dispatch
>> new session MonSession: client.? 10.177.32.66:0/25020608 is open for
>> client.? 10.177.32.66:0/25020608
>> 2011-12-01 20:18:17.143125 7fe03223b700 mon.0@0(leader).auth v146
>> preprocess_query auth(proto 0 21 bytes) v1 from client.?
>> 10.177.32.66:0/25020608
>> 2011-12-01 20:18:17.143168 7fe03223b700 -- 10.177.64.4:6789/0 -->
>> 10.177.32.66:0/25020608 -- auth_reply(proto 0 -95 Operation not
>> supported) v1 -- ?+0 0x2f30600 con 0x297adc0
>
>
> Here you can see qemu isn't authenticating correctly, probably because the
> rados user isn't being set by libvirt. With your libvirt version, you'll
> need to change:
>
>
> <source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
>
> to
>
> <source protocol='rbd' name='rbd:kvmtest/kvmtest1:id=foo'>
>
> to authenticate as client.foo. Any ceph config option can be set like that,
> separated by colons.
>
> If this still doesn't work, I'd suggest setting
> debug_rbd=20:debug_monc=20:debug_auth=20:log_to_stderr=2 to get more info in
> the libvirt instance log (usually /var/log/libvirt/qemu/instance_name.log).
> There may be apparmour restrictions preventing qemu from reading the keyring
> or ceph config files.
>
>
>> ceph.conf, and keyring.bin exists on server, and machine with qemu-kvm
>> in same directory:
>> ; global
>> [global]
>> ; enable secure authentication
>>
>> auth supported = cephx
>> keyring = /etc/ceph/keyring.bin
>>
>> debug rgw = 1
>>
>> rgw print continue = false
>> rgw socket path = /var/run/radosgw.sock
>>
>> ; monitors
>> ; You need at least one. You need at least three if you want to
>> ; tolerate any node failures. Always create an odd number.
>> [mon]
>> mon data = /vol0/data/mon.$id
>>
>> ; some minimal logging (just message traffic) to aid debugging
>>
>> debug ms = 1 ; see message traffic
>> debug mon = 0 ; monitor
>> debug paxos = 0 ; monitor replication
>> debug auth = 0 ;
>>
>> mon allowed clock drift = 2
>>
>> [mon.0]
>> host = 10-177-64-4
>> mon addr = 10.177.64.4:6789
>>
>> ; radosgw client list
>> [client.radosgw.10-177-64-4]
>>
>> host = 10-177-64-4
>> log file = /var/log/ceph/$name.log
>> debug rgw = 1
>> debug ms = 1
>>
>> ; osd
>> ; You need at least one. Two if you want data to be replicated.
>> ; Define as many as you like.
>> [osd]
>> ; This is where the btrfs volume will be mounted.
>>
>> osd data = /vol0/data/osd.$id
>>
>> ; Ideally, make this a separate disk or partition. A few GB
>> ; is usually enough; more if you have fast disks. You can use
>> ; a file under the osd data dir if need be
>> ; (e.g. /data/osd$id/journal), but it will be slower than a
>> ; separate disk or partition.
>>
>> osd journal = /vol0/data/osd.$id/journal
>>
>> ; If the OSD journal is a file, you need to specify the size.
>> This is specified in MB.
>>
>> osd journal size = 512
>>
>> filestore journal writeahead = 1
>> osd heartbeat grace = 5
>>
>> debug ms = 0 ; message traffic
>> debug osd = 0
>> debug filestore = 0 ; local object storage
>> debug journal = 0 ; local journaling
>> debug monc = 0
>> debug rados = 1
>>
>> [osd.0]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.0
>> keyring = /vol0/data/osd.0/keyring
>>
>> [osd.1]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.1
>> keyring = /vol0/data/osd.1/keyring
>>
>> [osd.2]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.2
>> keyring = /vol0/data/osd.2/keyring
>>
>> [osd.3]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.3
>> keyring = /vol0/data/osd.3/keyring
>>
>> [osd.4]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.4
>> keyring = /vol0/data/osd.4/keyring
>>
>> [osd.5]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.5
>> keyring = /vol0/data/osd.5/keyring
>>
>> [osd.6]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.6
>> keyring = /vol0/data/osd.6/keyring
>>
>> [osd.7]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.7
>> keyring = /vol0/data/osd.7/keyring
>>
>> [osd.8]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.8
>> keyring = /vol0/data/osd.8/keyring
>>
>> [osd.9]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.9
>> keyring = /vol0/data/osd.9/keyring
>>
>> [osd.10]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.10
>> keyring = /vol0/data/osd.10/keyring
>>
>> [osd.11]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.11
>> keyring = /vol0/data/osd.11/keyring
>>
>> [osd.12]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.12
>> keyring = /vol0/data/osd.12/keyring
>>
>> [osd.13]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.13
>> keyring = /vol0/data/osd.13/keyring
>>
>> [osd.14]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.14
>> keyring = /vol0/data/osd.14/keyring
>>
>> [osd.15]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.15
>> keyring = /vol0/data/osd.15/keyring
>>
>> [osd.16]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.16
>> keyring = /vol0/data/osd.16/keyring
>>
>> [osd.17]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.17
>> keyring = /vol0/data/osd.17/keyring
>>
>> [osd.18]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.18
>> keyring = /vol0/data/osd.18/keyring
>>
>> [osd.19]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.19
>> keyring = /vol0/data/osd.19/keyring
>>
>> [osd.20]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.20
>> keyring = /vol0/data/osd.20/keyring
>>
>> [osd.21]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.21
>> keyring = /vol0/data/osd.21/keyring
>>
>> [osd.22]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.22
>> keyring = /vol0/data/osd.22/keyring
>>
>> [osd.23]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.23
>> keyring = /vol0/data/osd.23/keyring
>>
>> [osd.24]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.24
>> keyring = /vol0/data/osd.24/keyring
>>
>> [osd.25]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.25
>> keyring = /vol0/data/osd.25/keyring
>>
>>
>
--
-----
Pozdrawiam
Sławek "sZiBis" Skowron
[-- Attachment #2: mon.0.log --]
[-- Type: application/octet-stream, Size: 12576 bytes --]
2011-12-01 23:27:42.275210 7fe61715f700 mon.0@0(leader) e1 ms_verify_authorizer 10.177.32.66:0/1006210 client protocol 0
2011-12-01 23:27:42.275705 7fe61b090700 -- 10.177.64.4:6789/0 <== client.? 10.177.32.66:0/1006210 1 ==== auth(proto 0 30 bytes) v1 ==== 56+0+0 (625540289 0 0) 0x1133000 con 0x1451500
2011-12-01 23:27:42.275728 7fe61b090700 mon.0@0(leader) e1 have connection
2011-12-01 23:27:42.275737 7fe61b090700 mon.0@0(leader) e1 do not have session, making new one
2011-12-01 23:27:42.275751 7fe61b090700 mon.0@0(leader) e1 ms_dispatch new session MonSession: client.? 10.177.32.66:0/1006210 is open for client.? 10.177.32.66:0/1006210
2011-12-01 23:27:42.275758 7fe61b090700 mon.0@0(leader) e1 setting timeout on session
2011-12-01 23:27:42.275765 7fe61b090700 mon.0@0(leader) e1 caps
2011-12-01 23:27:42.275773 7fe61b090700 mon.0@0(leader).auth v155 AuthMonitor::update_from_paxos()
2011-12-01 23:27:42.275783 7fe61b090700 mon.0@0(leader).auth v155 preprocess_query auth(proto 0 30 bytes) v1 from client.? 10.177.32.66:0/1006210
2011-12-01 23:27:42.275791 7fe61b090700 mon.0@0(leader).auth v155 prep_auth() blob_size=30
2011-12-01 23:27:42.275816 7fe61b090700 mon.0@0(leader).auth v155 AuthMonitor::assign_global_id m=auth(proto 0 30 bytes) v1 mon=0/1 last_allocated=5705 max_global_id=5796
2011-12-01 23:27:42.275825 7fe61b090700 mon.0@0(leader).auth v155 next_global_id should be 5706
2011-12-01 23:27:42.275853 7fe61b090700 cephx server client.admin: start_session server_challenge f5f193f5a09ea059
2011-12-01 23:27:42.275894 7fe61b090700 -- 10.177.64.4:6789/0 --> 10.177.32.66:0/1006210 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x11ec400 con 0x1451500
2011-12-01 23:27:42.276737 7fe61b090700 -- 10.177.64.4:6789/0 <== client.? 10.177.32.66:0/1006210 2 ==== auth(proto 2 32 bytes) v1 ==== 58+0+0 (2507233954 0 0) 0x11eca00 con 0x1451500
2011-12-01 23:27:42.276758 7fe61b090700 mon.0@0(leader) e1 have connection
2011-12-01 23:27:42.276771 7fe61b090700 mon.0@0(leader) e1 ms_dispatch existing session MonSession: client.? 10.177.32.66:0/1006210 is open for client.? 10.177.32.66:0/1006210
2011-12-01 23:27:42.276778 7fe61b090700 mon.0@0(leader) e1 caps
2011-12-01 23:27:42.276785 7fe61b090700 mon.0@0(leader).auth v155 AuthMonitor::update_from_paxos()
2011-12-01 23:27:42.276795 7fe61b090700 mon.0@0(leader).auth v155 preprocess_query auth(proto 2 32 bytes) v1 from client.? 10.177.32.66:0/1006210
2011-12-01 23:27:42.276802 7fe61b090700 mon.0@0(leader).auth v155 prep_auth() blob_size=32
2011-12-01 23:27:42.276808 7fe61b090700 cephx server client.admin: handle_request get_auth_session_key for client.admin
2011-12-01 23:27:42.276853 7fe61b090700 cephx server client.admin: checking key: req.key=b43f0fb4e2d728f0 expected_key=b43f0fb4e2d728f0
2011-12-01 23:27:42.276907 7fe61b090700 cephx keyserverdata: get_service_secret service auth id 10 AQA4vtZOwAMNKhAAWCkx2HemrXFSc/elGBTgiQ== expires 2011-12-02 00:37:28.705459
2011-12-01 23:27:42.276922 7fe61b090700 cephx: build_service_ticket_reply encoding 1 tickets with secret AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog==
2011-12-01 23:27:42.276945 7fe61b090700 cephx: build_service_ticket service auth secret_id 10 ticket_info.ticket.name=client.admin
2011-12-01 23:27:42.276970 7fe61b090700 cephx: service_ticket_blob is 0000 : 01 0a 00 00 00 00 00 00 00 60 00 00 00 3b bb d3 : .........`...;..
2011-12-01 23:27:42.276983 010 : ac db e9 74 da cd 3f e4 9b 8d c8 1f a1 37 6b dc : ...t..?......7k.
2011-12-01 23:27:42.276993 020 : 87 91 e9 96 f8 1a 67 1b 31 03 99 7f 08 5e 36 87 : ......g.1....^6.
2011-12-01 23:27:42.277002 030 : 70 12 4f 4d ff 53 df af 92 18 47 b9 36 d1 8c c9 : p.OM.S....G.6...
2011-12-01 23:27:42.277011 040 : e9 6f b8 f3 79 59 8d 20 90 da 73 c7 c4 f9 2d 62 : .o..yY. ..s...-b
2011-12-01 23:27:42.277019 050 : fe 68 c2 cd e9 02 22 c6 b2 e8 bc 3b e1 fd 89 ab : .h...."....;....
2011-12-01 23:27:42.277027 060 : 06 6d ad b1 46 55 c5 4f 3f 74 a8 42 d8 : .m..FU.O?t.B.
2011-12-01 23:27:42.277032 2011-12-01 23:27:42.277040 7fe61b090700 cephx keyserverdata: get_caps: name=client.admin
2011-12-01 23:27:42.277047 7fe61b090700 cephx keyserverdata: get_secret: num of caps=3
2011-12-01 23:27:42.277072 7fe61b090700 -- 10.177.64.4:6789/0 --> 10.177.32.66:0/1006210 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x1133000 con 0x1451500
2011-12-01 23:27:42.277675 7fe61b090700 -- 10.177.64.4:6789/0 <== client.? 10.177.32.66:0/1006210 3 ==== auth(proto 2 165 bytes) v1 ==== 191+0+0 (3885384660 0 0) 0x11ec800 con 0x1451500
2011-12-01 23:27:42.277692 7fe61b090700 mon.0@0(leader) e1 have connection
2011-12-01 23:27:42.277705 7fe61b090700 mon.0@0(leader) e1 ms_dispatch existing session MonSession: client.? 10.177.32.66:0/1006210 is openallow * for client.? 10.177.32.66:0/1006210
2011-12-01 23:27:42.277711 7fe61b090700 mon.0@0(leader) e1 caps allow *
2011-12-01 23:27:42.277718 7fe61b090700 mon.0@0(leader).auth v155 AuthMonitor::update_from_paxos()
2011-12-01 23:27:42.277728 7fe61b090700 mon.0@0(leader).auth v155 preprocess_query auth(proto 2 165 bytes) v1 from client.? 10.177.32.66:0/1006210
2011-12-01 23:27:42.277734 7fe61b090700 mon.0@0(leader).auth v155 prep_auth() blob_size=165
2011-12-01 23:27:42.277741 7fe61b090700 cephx server client.admin: handle_request get_principal_session_key
2011-12-01 23:27:42.277749 7fe61b090700 cephx: verify_authorizer decrypted service auth secret_id=10
2011-12-01 23:27:42.277783 7fe61b090700 cephx: verify_authorizer global_id=5706
2011-12-01 23:27:42.277811 7fe61b090700 cephx: verify_authorizer ok nonce 170deb1275cd21be reply_bl.length()=36
2011-12-01 23:27:42.277821 7fe61b090700 cephx server client.admin: ticket_req.keys = 5
2011-12-01 23:27:42.277836 7fe61b090700 cephx server client.admin: adding key for service mon
2011-12-01 23:27:42.277852 7fe61b090700 cephx keyserverdata: get_service_secret service mon id 119 AQCu5ddOuFssLhAA6Q3WtNQagL3YEdbHtFC94Q== expires 2011-12-01 23:38:06.774620
2011-12-01 23:27:42.277876 7fe61b090700 cephx keyserverdata: get_caps: name=client.admin
2011-12-01 23:27:42.277885 7fe61b090700 cephx keyserverdata: get_secret: num of caps=3
2011-12-01 23:27:42.277895 7fe61b090700 cephx server client.admin: adding key for service osd
2011-12-01 23:27:42.277907 7fe61b090700 cephx keyserverdata: get_service_secret service osd id 119 AQCu5ddOoNwsLhAA0e0LIoq04CBaYl1wREOhDw== expires 2011-12-01 23:38:06.774681
2011-12-01 23:27:42.277926 7fe61b090700 cephx keyserverdata: get_caps: name=client.admin
2011-12-01 23:27:42.277935 7fe61b090700 cephx keyserverdata: get_secret: num of caps=3
2011-12-01 23:27:42.277949 7fe61b090700 cephx: build_service_ticket_reply encoding 2 tickets with secret AQBe/9dOGBWBEBAAPxLkRBzsa1tq2hnN8crp9Q==
2011-12-01 23:27:42.277970 7fe61b090700 cephx: build_service_ticket service mon secret_id 119 ticket_info.ticket.name=client.admin
2011-12-01 23:27:42.277995 7fe61b090700 cephx: service_ticket_blob is 0000 : 01 77 00 00 00 00 00 00 00 70 00 00 00 e4 96 34 : .w.......p.....4
2011-12-01 23:27:42.278007 010 : ce 7a df bb e1 fd 7d 78 f2 9f 91 92 f7 6d 63 d3 : .z....}x.....mc.
2011-12-01 23:27:42.278016 020 : 8a 60 de 06 19 df d1 b6 25 28 d8 f2 9c 3b 99 67 : .`......%(...;.g
2011-12-01 23:27:42.278025 030 : ad 2d b4 08 d9 42 31 70 e6 68 03 c0 c4 ea 47 ea : .-...B1p.h....G.
2011-12-01 23:27:42.278034 040 : 5a 79 7a 49 74 57 b0 bb 4d 47 9e 29 cd f9 09 40 : ZyzItW..MG.)...@
2011-12-01 23:27:42.278043 050 : 4d e7 8e 9a d8 c6 63 eb 7e 78 f4 22 97 97 79 d5 : M.....c.~x."..y.
2011-12-01 23:27:42.278051 060 : a6 d2 ee cf 76 f9 78 66 5e 75 f9 83 5e bc 9e ff : ....v.xf^u..^...
2011-12-01 23:27:42.278059 070 : 56 5c af 76 70 62 4a 53 96 7c 2e 54 ca : V\.vpbJS.|.T.
2011-12-01 23:27:42.278065 2011-12-01 23:27:42.278081 7fe61b090700 cephx: build_service_ticket service osd secret_id 119 ticket_info.ticket.name=client.admin
2011-12-01 23:27:42.278105 7fe61b090700 cephx: service_ticket_blob is 0000 : 01 77 00 00 00 00 00 00 00 70 00 00 00 52 bc 57 : .w.......p...R.W
2011-12-01 23:27:42.278115 010 : 7f fd b6 71 10 5a 15 47 79 53 ba a0 02 5b 52 05 : ...q.Z.GyS...[R.
2011-12-01 23:27:42.278124 020 : 4d 7b b4 c6 f3 47 cb 6f 1b 17 85 8d c1 c6 7d d2 : M{...G.o......}.
2011-12-01 23:27:42.278133 030 : 96 5e d0 c3 6a 87 28 c1 cd 5f 02 14 16 6c a7 2d : .^..j.(.._...l.-
2011-12-01 23:27:42.278141 040 : 3c fd 11 05 9c d2 6e 13 d6 f4 d8 f5 c6 54 0f db : <.....n......T..
2011-12-01 23:27:42.278150 050 : d0 65 c6 b6 fb 5f f1 15 58 62 dc ed f5 d3 9b 80 : .e..._..Xb......
2011-12-01 23:27:42.278158 060 : 5d 0b 6c 47 8e 97 59 87 dc 1b f7 b7 1b 66 fd 63 : ].lG..Y......f.c
2011-12-01 23:27:42.278168 070 : f2 d2 c4 08 a6 88 47 bf f6 d2 4e 29 cd : ......G...N).
2011-12-01 23:27:42.278173 2011-12-01 23:27:42.278192 7fe61b090700 -- 10.177.64.4:6789/0 --> 10.177.32.66:0/1006210 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x11eca00 con 0x1451500
2011-12-01 23:27:42.278719 7fe61b090700 -- 10.177.64.4:6789/0 <== client.? 10.177.32.66:0/1006210 4 ==== mon_subscribe({monmap=0+}) v2 ==== 23+0+0 (1620593354 0 0) 0x120a000 con 0x1451500
2011-12-01 23:27:42.278740 7fe61b090700 mon.0@0(leader) e1 have connection
2011-12-01 23:27:42.278760 7fe61b090700 mon.0@0(leader) e1 ms_dispatch existing session MonSession: client.? 10.177.32.66:0/1006210 is openallow * for client.? 10.177.32.66:0/1006210
2011-12-01 23:27:42.278773 7fe61b090700 mon.0@0(leader) e1 caps allow *
2011-12-01 23:27:42.278786 7fe61b090700 mon.0@0(leader) e1 handle_subscribe mon_subscribe({monmap=0+}) v2
2011-12-01 23:27:42.278805 7fe61b090700 mon.0@0(leader) e1 check_sub monmap next 0 have 1
2011-12-01 23:27:42.278830 7fe61b090700 -- 10.177.64.4:6789/0 --> 10.177.32.66:0/1006210 -- mon_map v1 -- ?+0 0x13e7000 con 0x1451500
2011-12-01 23:27:42.278857 7fe61b090700 -- 10.177.64.4:6789/0 --> client.? 10.177.32.66:0/1006210 -- mon_subscribe_ack(300s) v1 -- ?+0 0x11bd900
2011-12-01 23:27:42.278877 7fe61b090700 -- 10.177.64.4:6789/0 <== client.5706 10.177.32.66:0/1006210 5 ==== mon_subscribe({monmap=0+,osdmap=0}) v2 ==== 42+0+0 (982583713 0 0) 0x14851c0 con 0x1451500
2011-12-01 23:27:42.278889 7fe61b090700 mon.0@0(leader) e1 have connection
2011-12-01 23:27:42.278902 7fe61b090700 mon.0@0(leader) e1 ms_dispatch existing session MonSession: client.? 10.177.32.66:0/1006210 is openallow * for client.? 10.177.32.66:0/1006210
2011-12-01 23:27:42.278909 7fe61b090700 mon.0@0(leader) e1 caps allow *
2011-12-01 23:27:42.278949 7fe61b090700 mon.0@0(leader) e1 handle_subscribe mon_subscribe({monmap=0+,osdmap=0}) v2
2011-12-01 23:27:42.278965 7fe61b090700 mon.0@0(leader) e1 check_sub monmap next 0 have 1
2011-12-01 23:27:42.278980 7fe61b090700 -- 10.177.64.4:6789/0 --> 10.177.32.66:0/1006210 -- mon_map v1 -- ?+0 0x120a000 con 0x1451500
2011-12-01 23:27:42.279116 7fe61b090700 -- 10.177.64.4:6789/0 --> client.? 10.177.32.66:0/1006210 -- osd_map(269..269 src has 1..269) v1 -- ?+0 0x11ec800
2011-12-01 23:27:42.279143 7fe61b090700 -- 10.177.64.4:6789/0 --> client.5706 10.177.32.66:0/1006210 -- mon_subscribe_ack(300s) v1 -- ?+0 0x11bd780
2011-12-01 23:27:42.279163 7fe61b090700 -- 10.177.64.4:6789/0 <== client.5706 10.177.32.66:0/1006210 6 ==== mon_subscribe({monmap=0+,osdmap=0}) v2 ==== 42+0+0 (982583713 0 0) 0x1485000 con 0x1451500
2011-12-01 23:27:42.279171 7fe61b090700 mon.0@0(leader) e1 have connection
2011-12-01 23:27:42.279182 7fe61b090700 mon.0@0(leader) e1 ms_dispatch existing session MonSession: client.? 10.177.32.66:0/1006210 is openallow * for client.? 10.177.32.66:0/1006210
2011-12-01 23:27:42.279189 7fe61b090700 mon.0@0(leader) e1 caps allow *
2011-12-01 23:27:42.279197 7fe61b090700 mon.0@0(leader) e1 handle_subscribe mon_subscribe({monmap=0+,osdmap=0}) v2
2011-12-01 23:27:42.279208 7fe61b090700 mon.0@0(leader) e1 check_sub monmap next 0 have 1
2011-12-01 23:27:42.279226 7fe61b090700 -- 10.177.64.4:6789/0 --> 10.177.32.66:0/1006210 -- mon_map v1 -- ?+0 0x14851c0 con 0x1451500
2011-12-01 23:27:42.279317 7fe61b090700 -- 10.177.64.4:6789/0 --> client.? 10.177.32.66:0/1006210 -- osd_map(269..269 src has 1..269) v1 -- ?+0 0x11ec600
2011-12-01 23:27:42.279338 7fe61b090700 -- 10.177.64.4:6789/0 --> client.5706 10.177.32.66:0/1006210 -- mon_subscribe_ack(300s) v1 -- ?+0 0x11bd600
2011-12-01 23:27:42.294278 7fe61b090700 mon.0@0(leader) e1 ms_handle_reset 0x1451500 10.177.32.66:0/1006210
2011-12-01 23:27:42.294305 7fe61b090700 mon.0@0(leader) e1 reset/close on session client.? 10.177.32.66:0/1006210
2011-12-01 23:27:42.294318 7fe61b090700 mon.0@0(leader) e1 remove_session MonSession: client.? 10.177.32.66:0/1006210 is openallow * client.? 10.177.32.66:0/1006210
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Problem with attaching rbd device in qemu-kvm
2011-12-09 12:48 ` Sławomir Skowron
@ 2011-12-12 18:53 ` Josh Durgin
2011-12-13 12:56 ` Sławomir Skowron
0 siblings, 1 reply; 10+ messages in thread
From: Josh Durgin @ 2011-12-12 18:53 UTC (permalink / raw)
To: Sławomir Skowron; +Cc: ceph-devel
On 12/09/2011 04:48 AM, Sławomir Skowron wrote:
> Sorry, for my lag, but i was sick.
>
> I handled problem with apparmor before and now it's not a problem,
> even when i send mail before, it was solved.
>
> Ok when i create image from qemu-img it's looks like that, and works
> perfect, (attachment with log of this opperation.)
>
> But when i use virsh, it's still a problem. Just like he don't know
> what user is going to be auth.
>
> I try, as client.admin (default i hope :)) only with id=admin in
> kvm.xml, or client=admin, and other combinations - like client.rbd
> etc., Still nothing.
admin is the default, and the correct option for client.admin is
id=admin, or id=foo for client.foo.
>
> I try, improve a keyring file, adding client rbd into keyring on
> machine with kvm hypervisor in /etc/ceph/keyring.bin
>
> [client.rbd]
> key = AAAAAAAAAAAAAAAA
> auid = 18446744073709551615
> caps mds = "allow"
> caps mon = "allow r"
> caps osd = "allow rw pool=kvmtest"
>
> and on cluster site like above. I was added, a section [client.rbd]
> with keyring path, and it's still a problem.
>
Did you run 'ceph auth add -i /etc/ceph/keyring.bin'?
Make sure 'ceph auth list' shows client.rbd with the right key.
> 2011-12-09 13:29:11.519096 7fe616b59700 mon.0@0(leader) e1
> ms_verify_authorizer 10.177.32.66:0/58020608 client protocol 0
> root@s3-10-177-64-4:~# tail -n10000 /var/log/ceph/mon.0.log | grep 13:29:11.
> 2011-12-09 13:29:11.519096 7fe616b59700 mon.0@0(leader) e1
> ms_verify_authorizer 10.177.32.66:0/58020608 client protocol 0
> 2011-12-09 13:29:11.519624 7fe61b090700 -- 10.177.64.4:6789/0<==
> client.? 10.177.32.66:0/58020608 1 ==== auth(proto 0 21 bytes) v1 ====
> 47+0+0 (3310669357 0 0) 0x17f2e00 con 0x19d2c80
> 2011-12-09 13:29:11.519645 7fe61b090700 mon.0@0(leader) e1 have connection
> 2011-12-09 13:29:11.519655 7fe61b090700 mon.0@0(leader) e1 do not have
> session, making new one
> 2011-12-09 13:29:11.519668 7fe61b090700 mon.0@0(leader) e1 ms_dispatch
> new session MonSession: client.? 10.177.32.66:0/58020608 is open for
> client.? 10.177.32.66:0/58020608
> 2011-12-09 13:29:11.519674 7fe61b090700 mon.0@0(leader) e1 setting
> timeout on session
> 2011-12-09 13:29:11.519680 7fe61b090700 mon.0@0(leader) e1 caps
> 2011-12-09 13:29:11.519688 7fe61b090700 mon.0@0(leader).auth v353
> AuthMonitor::update_from_paxos()
> 2011-12-09 13:29:11.519697 7fe61b090700 mon.0@0(leader).auth v353
> preprocess_query auth(proto 0 21 bytes) v1 from client.?
> 10.177.32.66:0/58020608
> 2011-12-09 13:29:11.519703 7fe61b090700 mon.0@0(leader).auth v353
> prep_auth() blob_size=21
> 2011-12-09 13:29:11.519735 7fe61b090700 -- 10.177.64.4:6789/0 -->
> 10.177.32.66:0/58020608 -- auth_reply(proto 0 -95 Operation not
> supported) v1 -- ?+0 0x1246200 con 0x19d2c80
>
> from libvirt.log
>
> 23:20:56.689: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
> : cannot resolve symlink rbd:kvmtest/kvmtest1:name=admin: No such file
> or directory
> 23:20:56.882: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
> Unable to restore security label on rbd:kvmtest/kvmtest1:name=admin
> 23:21:09.172: 1309: error : qemuMonitorTextAddDrive:2418 : operation
> failed: open disk image file failed
> 23:21:09.172: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
> : cannot resolve symlink rbd:kvmtest/kvmtest1:name=admin: No such file
> or directory
> 23:21:09.364: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
> Unable to restore security label on rbd:kvmtest/kvmtest1:name=admin
> 23:21:49.645: 1309: error : qemuMonitorTextAddDrive:2418 : operation
> failed: open disk image file failed
> 23:21:49.645: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
> : cannot resolve symlink rbd:kvmtest/kvmtest1:id=admin: No such file
> or directory
> 23:21:49.853: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
> Unable to restore security label on rbd:kvmtest/kvmtest1:id=admin
> 13:08:08.924: 1309: error : qemuMonitorTextAddDrive:2418 : operation
> failed: open disk image file failed
> 13:08:08.924: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
> : cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or
> directory
> 13:08:09.117: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
> Unable to restore security label on rbd:kvmtest/kvmtest1
> 13:09:45.402: 1309: error : qemuMonitorTextAddDrive:2418 : operation
> failed: open disk image file failed
> 13:29:05.554: 1310: error : qemuMonitorTextAddDrive:2418 : operation
> failed: open disk image file failed
> 13:29:05.554: 1310: error : virSecurityDACRestoreSecurityFileLabel:143
> : cannot resolve symlink kvmtest/kvmtest1: No such file or directory
> 13:29:05.751: 1310: warning : qemuDomainAttachPciDiskDevice:250 :
> Unable to restore security label on kvmtest/kvmtest1
> 13:29:05.751: 1310: warning : virEventPollUpdateHandle:147 : Ignoring
> invalid update watch -1
This is libvirt mistakenly trying to treat the rbd disk as a file -
fixed in 20e1233c31e3150d259073c523077acdccb5d419 to libvirt. If you
don't want to upgrade libvirt, someone on IRC last week reported that
installing libapparmor1 fixed this problem for them, since libvirt no
longer fell back to it's own DAC implementation.
> 2011/12/1 Josh Durgin<josh.durgin@dreamhost.com>:
>> On 12/01/2011 11:37 AM, Sławomir Skowron wrote:
>>>
>>> I have some problems. Can you help me ??
>>>
>>> ceph cluster:
>>> ceph 0.38, oneiric, kernel 3.0.0 x86_64 - now only one machine.
>>>
>>> kvm hypervisor:
>>> kvm version 1.0-rc4 (0.15.92), libvirt 0.9.2, kernel 3.0.0, Ubuntu
>>> oneiric x86_64.
>>>
>>> I create image from qemu-img on machine with kvm VM's, and it works very
>>> well:
>>>
>>> qemu-img create -f rbd rbd:kvmtest/kvmtest1 10G
>>> Formatting 'rbd:kvmtest/kvmtest1', fmt=rbd size=10737418240 cluster_size=0
>>>
>>> in ceph cluster machines:
>>>
>>> rados -p kvmtest ls
>>> kvmtest1.rbd
>>> rbd_directory
>>> rbd_info
>>>
>>> But when i try to attach new device to kvm VM like this:
>>>
>>> virsh attach-device one-10 /tmp/kvm.xml
>>>
>>> with kvm.xml like this:
>>>
>>> <disk type='network' device='disk'>
>>> <driver name='qemu' type='raw' cache='writeback'/>
>>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
>>> <host name='10.177.64.4' port='6789'/>
>>> </source>
>>> <target dev='vdc' bus='virtio'/>
>>> </disk>
>>>
>>> output of virsh:
>>>
>>> error: Failed to attach device from /tmp/kvm.xml
>>> error: cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or
>>> directory
>>
>>
>> From the monitor output below, I'm assuming this is just a bad error message
>> from libvirt.
>>
>>
>>> and mon.0 log in ceph cluster.
>>>
>>> 2011-12-01 20:18:17.142595 7fe02dd04700 mon.0@0(leader) e1
>>> ms_verify_authorizer 10.177.32.66:0/25020608 client protocol 0
>>> 2011-12-01 20:18:17.143041 7fe03223b700 -- 10.177.64.4:6789/0<==
>>> client.? 10.177.32.66:0/25020608 1 ==== auth(proto 0 21 bytes) v1 ====
>>> 47+0+0 (3310669357 0 0) 0x29f6a00 con 0x297adc0
>>> 2011-12-01 20:18:17.143095 7fe03223b700 mon.0@0(leader) e1 ms_dispatch
>>> new session MonSession: client.? 10.177.32.66:0/25020608 is open for
>>> client.? 10.177.32.66:0/25020608
>>> 2011-12-01 20:18:17.143125 7fe03223b700 mon.0@0(leader).auth v146
>>> preprocess_query auth(proto 0 21 bytes) v1 from client.?
>>> 10.177.32.66:0/25020608
>>> 2011-12-01 20:18:17.143168 7fe03223b700 -- 10.177.64.4:6789/0 -->
>>> 10.177.32.66:0/25020608 -- auth_reply(proto 0 -95 Operation not
>>> supported) v1 -- ?+0 0x2f30600 con 0x297adc0
>>
>>
>> Here you can see qemu isn't authenticating correctly, probably because the
>> rados user isn't being set by libvirt. With your libvirt version, you'll
>> need to change:
>>
>>
>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
>>
>> to
>>
>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1:id=foo'>
>>
>> to authenticate as client.foo. Any ceph config option can be set like that,
>> separated by colons.
>>
>> If this still doesn't work, I'd suggest setting
>> debug_rbd=20:debug_monc=20:debug_auth=20:log_to_stderr=2 to get more info in
>> the libvirt instance log (usually /var/log/libvirt/qemu/instance_name.log).
>> There may be apparmour restrictions preventing qemu from reading the keyring
>> or ceph config files.
>>
>>
>>> ceph.conf, and keyring.bin exists on server, and machine with qemu-kvm
>>> in same directory:
>>> ; global
>>> [global]
>>> ; enable secure authentication
>>>
>>> auth supported = cephx
>>> keyring = /etc/ceph/keyring.bin
>>>
>>> debug rgw = 1
>>>
>>> rgw print continue = false
>>> rgw socket path = /var/run/radosgw.sock
>>>
>>> ; monitors
>>> ; You need at least one. You need at least three if you want to
>>> ; tolerate any node failures. Always create an odd number.
>>> [mon]
>>> mon data = /vol0/data/mon.$id
>>>
>>> ; some minimal logging (just message traffic) to aid debugging
>>>
>>> debug ms = 1 ; see message traffic
>>> debug mon = 0 ; monitor
>>> debug paxos = 0 ; monitor replication
>>> debug auth = 0 ;
>>>
>>> mon allowed clock drift = 2
>>>
>>> [mon.0]
>>> host = 10-177-64-4
>>> mon addr = 10.177.64.4:6789
>>>
>>> ; radosgw client list
>>> [client.radosgw.10-177-64-4]
>>>
>>> host = 10-177-64-4
>>> log file = /var/log/ceph/$name.log
>>> debug rgw = 1
>>> debug ms = 1
>>>
>>> ; osd
>>> ; You need at least one. Two if you want data to be replicated.
>>> ; Define as many as you like.
>>> [osd]
>>> ; This is where the btrfs volume will be mounted.
>>>
>>> osd data = /vol0/data/osd.$id
>>>
>>> ; Ideally, make this a separate disk or partition. A few GB
>>> ; is usually enough; more if you have fast disks. You can use
>>> ; a file under the osd data dir if need be
>>> ; (e.g. /data/osd$id/journal), but it will be slower than a
>>> ; separate disk or partition.
>>>
>>> osd journal = /vol0/data/osd.$id/journal
>>>
>>> ; If the OSD journal is a file, you need to specify the size.
>>> This is specified in MB.
>>>
>>> osd journal size = 512
>>>
>>> filestore journal writeahead = 1
>>> osd heartbeat grace = 5
>>>
>>> debug ms = 0 ; message traffic
>>> debug osd = 0
>>> debug filestore = 0 ; local object storage
>>> debug journal = 0 ; local journaling
>>> debug monc = 0
>>> debug rados = 1
>>>
>>> [osd.0]
>>> host = 10-177-64-4
>>> osd data = /vol0/data/osd.0
>>> keyring = /vol0/data/osd.0/keyring
>>>
>>> [osd.1]
>>> host = 10-177-64-4
>>> osd data = /vol0/data/osd.1
>>> keyring = /vol0/data/osd.1/keyring
>>>
>>> [osd.2]
>>> host = 10-177-64-4
>>> osd data = /vol0/data/osd.2
>>> keyring = /vol0/data/osd.2/keyring
>>>
>>> [osd.3]
>>> host = 10-177-64-4
>>> osd data = /vol0/data/osd.3
>>> keyring = /vol0/data/osd.3/keyring
>>>
>>> [osd.4]
>>> host = 10-177-64-4
>>> osd data = /vol0/data/osd.4
>>> keyring = /vol0/data/osd.4/keyring
>>>
>>> [osd.5]
>>> host = 10-177-64-4
>>> osd data = /vol0/data/osd.5
>>> keyring = /vol0/data/osd.5/keyring
>>>
>>> [osd.6]
>>> host = 10-177-64-4
>>> osd data = /vol0/data/osd.6
>>> keyring = /vol0/data/osd.6/keyring
>>>
>>> [osd.7]
>>> host = 10-177-64-4
>>> osd data = /vol0/data/osd.7
>>> keyring = /vol0/data/osd.7/keyring
>>>
>>> [osd.8]
>>> host = 10-177-64-4
>>> osd data = /vol0/data/osd.8
>>> keyring = /vol0/data/osd.8/keyring
>>>
>>> [osd.9]
>>> host = 10-177-64-4
>>> osd data = /vol0/data/osd.9
>>> keyring = /vol0/data/osd.9/keyring
>>>
>>> [osd.10]
>>> host = 10-177-64-4
>>> osd data = /vol0/data/osd.10
>>> keyring = /vol0/data/osd.10/keyring
>>>
>>> [osd.11]
>>> host = 10-177-64-4
>>> osd data = /vol0/data/osd.11
>>> keyring = /vol0/data/osd.11/keyring
>>>
>>> [osd.12]
>>> host = 10-177-64-4
>>> osd data = /vol0/data/osd.12
>>> keyring = /vol0/data/osd.12/keyring
>>>
>>> [osd.13]
>>> host = 10-177-64-4
>>> osd data = /vol0/data/osd.13
>>> keyring = /vol0/data/osd.13/keyring
>>>
>>> [osd.14]
>>> host = 10-177-64-4
>>> osd data = /vol0/data/osd.14
>>> keyring = /vol0/data/osd.14/keyring
>>>
>>> [osd.15]
>>> host = 10-177-64-4
>>> osd data = /vol0/data/osd.15
>>> keyring = /vol0/data/osd.15/keyring
>>>
>>> [osd.16]
>>> host = 10-177-64-4
>>> osd data = /vol0/data/osd.16
>>> keyring = /vol0/data/osd.16/keyring
>>>
>>> [osd.17]
>>> host = 10-177-64-4
>>> osd data = /vol0/data/osd.17
>>> keyring = /vol0/data/osd.17/keyring
>>>
>>> [osd.18]
>>> host = 10-177-64-4
>>> osd data = /vol0/data/osd.18
>>> keyring = /vol0/data/osd.18/keyring
>>>
>>> [osd.19]
>>> host = 10-177-64-4
>>> osd data = /vol0/data/osd.19
>>> keyring = /vol0/data/osd.19/keyring
>>>
>>> [osd.20]
>>> host = 10-177-64-4
>>> osd data = /vol0/data/osd.20
>>> keyring = /vol0/data/osd.20/keyring
>>>
>>> [osd.21]
>>> host = 10-177-64-4
>>> osd data = /vol0/data/osd.21
>>> keyring = /vol0/data/osd.21/keyring
>>>
>>> [osd.22]
>>> host = 10-177-64-4
>>> osd data = /vol0/data/osd.22
>>> keyring = /vol0/data/osd.22/keyring
>>>
>>> [osd.23]
>>> host = 10-177-64-4
>>> osd data = /vol0/data/osd.23
>>> keyring = /vol0/data/osd.23/keyring
>>>
>>> [osd.24]
>>> host = 10-177-64-4
>>> osd data = /vol0/data/osd.24
>>> keyring = /vol0/data/osd.24/keyring
>>>
>>> [osd.25]
>>> host = 10-177-64-4
>>> osd data = /vol0/data/osd.25
>>> keyring = /vol0/data/osd.25/keyring
>>>
>>>
>>
>
>
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Problem with attaching rbd device in qemu-kvm
2011-12-12 18:53 ` Josh Durgin
@ 2011-12-13 12:56 ` Sławomir Skowron
2011-12-13 20:08 ` Josh Durgin
0 siblings, 1 reply; 10+ messages in thread
From: Sławomir Skowron @ 2011-12-13 12:56 UTC (permalink / raw)
To: Josh Durgin; +Cc: ceph-devel
Finally, i manage the problem with rbd with kvm 1.0, and libvirt
0.9.8, or i think i manage :), but i get stuck with one thing after.
2011-12-13 12:13:31.173+0000: 21512: error :
qemuMonitorJSONCheckError:318 : internal error unable to execute QEMU
command 'device_add': Bus 'pci.0' does not support hotplugging
2011-12-13 12:13:31.173+0000: 21512: warning :
qemuDomainAttachPciDiskDevice:244 : qemuMonitorAddDevice failed on
file=rbd:rbd/testdysk:id=admin:key=AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog==:auth_supported=cephx
none:mon_host=10.177.64.4\:6789,if=none,id=drive-virtio-disk2,format=raw
(virtio-blk-pci,bus=pci.0,addr=0x7,drive=drive-virtio-disk2,id=virtio-disk2)
2011-12-13 12:13:31.173+0000: 21512: error :
virSecurityDACRestoreSecurityFileLabel:143 : cannot resolve symlink
rbd/testdysk: No such file or directory2011-12-13 12:13:31.173+0000:
21512: warning : qemuDomainAttachPciDiskDevice:287 : Unable to restore
security label on rbd/testdysk
and i can't find any info about that.
my kvm.xml inject file:
<disk type="network" device="disk">
<driver name="qemu" type="raw"/>
<auth username="admin">
<secret type="ceph"
uuid="0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f"/>
</auth>
<source protocol="rbd" name="rbd/testdysk">
<host name="10.177.64.4" port="6789"/>
</source>
<target dev="sdc" bus="virtio"/>
</disk>
secret:
cat /tmp/secret.xml
<secret ephemeral='no' private='no'>
<uuid>0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f</uuid>
<usage type='ceph'>
<domain>rbd/admin</domain>
<name>admin</name>
</usage>
</secret>
virsh secret-define /tmp/secret.xml
Secret 0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f created
virsh secret-set-value 0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f
AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog==
Secret value set
2011/12/12 Josh Durgin <josh.durgin@dreamhost.com>:
> On 12/09/2011 04:48 AM, Sławomir Skowron wrote:
>>
>> Sorry, for my lag, but i was sick.
>>
>> I handled problem with apparmor before and now it's not a problem,
>> even when i send mail before, it was solved.
>>
>> Ok when i create image from qemu-img it's looks like that, and works
>> perfect, (attachment with log of this opperation.)
>>
>> But when i use virsh, it's still a problem. Just like he don't know
>> what user is going to be auth.
>>
>> I try, as client.admin (default i hope :)) only with id=admin in
>> kvm.xml, or client=admin, and other combinations - like client.rbd
>> etc., Still nothing.
>
>
> admin is the default, and the correct option for client.admin is id=admin,
> or id=foo for client.foo.
>
>
>>
>> I try, improve a keyring file, adding client rbd into keyring on
>> machine with kvm hypervisor in /etc/ceph/keyring.bin
>>
>> [client.rbd]
>> key = AAAAAAAAAAAAAAAA
>> auid = 18446744073709551615
>> caps mds = "allow"
>> caps mon = "allow r"
>> caps osd = "allow rw pool=kvmtest"
>>
>> and on cluster site like above. I was added, a section [client.rbd]
>> with keyring path, and it's still a problem.
>>
>
> Did you run 'ceph auth add -i /etc/ceph/keyring.bin'?
> Make sure 'ceph auth list' shows client.rbd with the right key.
Overall i want to use admin id, and i think now it's correctly used in xml.
>
>
>> 2011-12-09 13:29:11.519096 7fe616b59700 mon.0@0(leader) e1
>> ms_verify_authorizer 10.177.32.66:0/58020608 client protocol 0
>> root@s3-10-177-64-4:~# tail -n10000 /var/log/ceph/mon.0.log | grep
>> 13:29:11.
>> 2011-12-09 13:29:11.519096 7fe616b59700 mon.0@0(leader) e1
>> ms_verify_authorizer 10.177.32.66:0/58020608 client protocol 0
>> 2011-12-09 13:29:11.519624 7fe61b090700 -- 10.177.64.4:6789/0<==
>> client.? 10.177.32.66:0/58020608 1 ==== auth(proto 0 21 bytes) v1 ====
>> 47+0+0 (3310669357 0 0) 0x17f2e00 con 0x19d2c80
>> 2011-12-09 13:29:11.519645 7fe61b090700 mon.0@0(leader) e1 have connection
>> 2011-12-09 13:29:11.519655 7fe61b090700 mon.0@0(leader) e1 do not have
>> session, making new one
>> 2011-12-09 13:29:11.519668 7fe61b090700 mon.0@0(leader) e1 ms_dispatch
>> new session MonSession: client.? 10.177.32.66:0/58020608 is open for
>> client.? 10.177.32.66:0/58020608
>> 2011-12-09 13:29:11.519674 7fe61b090700 mon.0@0(leader) e1 setting
>> timeout on session
>> 2011-12-09 13:29:11.519680 7fe61b090700 mon.0@0(leader) e1 caps
>> 2011-12-09 13:29:11.519688 7fe61b090700 mon.0@0(leader).auth v353
>> AuthMonitor::update_from_paxos()
>> 2011-12-09 13:29:11.519697 7fe61b090700 mon.0@0(leader).auth v353
>> preprocess_query auth(proto 0 21 bytes) v1 from client.?
>> 10.177.32.66:0/58020608
>> 2011-12-09 13:29:11.519703 7fe61b090700 mon.0@0(leader).auth v353
>> prep_auth() blob_size=21
>> 2011-12-09 13:29:11.519735 7fe61b090700 -- 10.177.64.4:6789/0 -->
>> 10.177.32.66:0/58020608 -- auth_reply(proto 0 -95 Operation not
>> supported) v1 -- ?+0 0x1246200 con 0x19d2c80
>>
>> from libvirt.log
>>
>> 23:20:56.689: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
>> : cannot resolve symlink rbd:kvmtest/kvmtest1:name=admin: No such file
>> or directory
>> 23:20:56.882: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>> Unable to restore security label on rbd:kvmtest/kvmtest1:name=admin
>> 23:21:09.172: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>> failed: open disk image file failed
>> 23:21:09.172: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
>> : cannot resolve symlink rbd:kvmtest/kvmtest1:name=admin: No such file
>> or directory
>> 23:21:09.364: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>> Unable to restore security label on rbd:kvmtest/kvmtest1:name=admin
>> 23:21:49.645: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>> failed: open disk image file failed
>> 23:21:49.645: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
>> : cannot resolve symlink rbd:kvmtest/kvmtest1:id=admin: No such file
>> or directory
>> 23:21:49.853: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>> Unable to restore security label on rbd:kvmtest/kvmtest1:id=admin
>> 13:08:08.924: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>> failed: open disk image file failed
>> 13:08:08.924: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
>> : cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or
>> directory
>> 13:08:09.117: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>> Unable to restore security label on rbd:kvmtest/kvmtest1
>> 13:09:45.402: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>> failed: open disk image file failed
>> 13:29:05.554: 1310: error : qemuMonitorTextAddDrive:2418 : operation
>> failed: open disk image file failed
>> 13:29:05.554: 1310: error : virSecurityDACRestoreSecurityFileLabel:143
>> : cannot resolve symlink kvmtest/kvmtest1: No such file or directory
>> 13:29:05.751: 1310: warning : qemuDomainAttachPciDiskDevice:250 :
>> Unable to restore security label on kvmtest/kvmtest1
>> 13:29:05.751: 1310: warning : virEventPollUpdateHandle:147 : Ignoring
>> invalid update watch -1
>
>
> This is libvirt mistakenly trying to treat the rbd disk as a file - fixed in
> 20e1233c31e3150d259073c523077acdccb5d419 to libvirt. If you don't want to
> upgrade libvirt, someone on IRC last week reported that installing
> libapparmor1 fixed this problem for them, since libvirt no longer fell back
> to it's own DAC implementation.
hmm is this commited to any stable version now ?? i have installed
libapparmor1,
ri libapparmor1 2.7.0~beta1+bzr1774-1ubuntu2
changehat AppArmor library
and i use newest stable libvirt 0.9.8.
>
>
>> 2011/12/1 Josh Durgin<josh.durgin@dreamhost.com>:
>>>
>>> On 12/01/2011 11:37 AM, Sławomir Skowron wrote:
>>>>
>>>>
>>>> I have some problems. Can you help me ??
>>>>
>>>> ceph cluster:
>>>> ceph 0.38, oneiric, kernel 3.0.0 x86_64 - now only one machine.
>>>>
>>>> kvm hypervisor:
>>>> kvm version 1.0-rc4 (0.15.92), libvirt 0.9.2, kernel 3.0.0, Ubuntu
>>>> oneiric x86_64.
>>>>
>>>> I create image from qemu-img on machine with kvm VM's, and it works very
>>>> well:
>>>>
>>>> qemu-img create -f rbd rbd:kvmtest/kvmtest1 10G
>>>> Formatting 'rbd:kvmtest/kvmtest1', fmt=rbd size=10737418240
>>>> cluster_size=0
>>>>
>>>> in ceph cluster machines:
>>>>
>>>> rados -p kvmtest ls
>>>> kvmtest1.rbd
>>>> rbd_directory
>>>> rbd_info
>>>>
>>>> But when i try to attach new device to kvm VM like this:
>>>>
>>>> virsh attach-device one-10 /tmp/kvm.xml
>>>>
>>>> with kvm.xml like this:
>>>>
>>>> <disk type='network' device='disk'>
>>>> <driver name='qemu' type='raw' cache='writeback'/>
>>>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
>>>> <host name='10.177.64.4' port='6789'/>
>>>> </source>
>>>> <target dev='vdc' bus='virtio'/>
>>>> </disk>
>>>>
>>>> output of virsh:
>>>>
>>>> error: Failed to attach device from /tmp/kvm.xml
>>>> error: cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or
>>>> directory
>>>
>>>
>>>
>>> From the monitor output below, I'm assuming this is just a bad error
>>> message
>>> from libvirt.
>>>
>>>
>>>> and mon.0 log in ceph cluster.
>>>>
>>>> 2011-12-01 20:18:17.142595 7fe02dd04700 mon.0@0(leader) e1
>>>> ms_verify_authorizer 10.177.32.66:0/25020608 client protocol 0
>>>> 2011-12-01 20:18:17.143041 7fe03223b700 -- 10.177.64.4:6789/0<==
>>>> client.? 10.177.32.66:0/25020608 1 ==== auth(proto 0 21 bytes) v1 ====
>>>> 47+0+0 (3310669357 0 0) 0x29f6a00 con 0x297adc0
>>>> 2011-12-01 20:18:17.143095 7fe03223b700 mon.0@0(leader) e1 ms_dispatch
>>>> new session MonSession: client.? 10.177.32.66:0/25020608 is open for
>>>> client.? 10.177.32.66:0/25020608
>>>> 2011-12-01 20:18:17.143125 7fe03223b700 mon.0@0(leader).auth v146
>>>> preprocess_query auth(proto 0 21 bytes) v1 from client.?
>>>> 10.177.32.66:0/25020608
>>>> 2011-12-01 20:18:17.143168 7fe03223b700 -- 10.177.64.4:6789/0 -->
>>>> 10.177.32.66:0/25020608 -- auth_reply(proto 0 -95 Operation not
>>>> supported) v1 -- ?+0 0x2f30600 con 0x297adc0
>>>
>>>
>>>
>>> Here you can see qemu isn't authenticating correctly, probably because
>>> the
>>> rados user isn't being set by libvirt. With your libvirt version, you'll
>>> need to change:
>>>
>>>
>>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
>>>
>>> to
>>>
>>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1:id=foo'>
>>>
>>> to authenticate as client.foo. Any ceph config option can be set like
>>> that,
>>> separated by colons.
>>>
>>> If this still doesn't work, I'd suggest setting
>>> debug_rbd=20:debug_monc=20:debug_auth=20:log_to_stderr=2 to get more info
>>> in
>>> the libvirt instance log (usually
>>> /var/log/libvirt/qemu/instance_name.log).
>>> There may be apparmour restrictions preventing qemu from reading the
>>> keyring
>>> or ceph config files.
>>>
>>>
>>>> ceph.conf, and keyring.bin exists on server, and machine with qemu-kvm
>>>> in same directory:
>>>> ; global
>>>> [global]
>>>> ; enable secure authentication
>>>>
>>>> auth supported = cephx
>>>> keyring = /etc/ceph/keyring.bin
>>>>
>>>> debug rgw = 1
>>>>
>>>> rgw print continue = false
>>>> rgw socket path = /var/run/radosgw.sock
>>>>
>>>> ; monitors
>>>> ; You need at least one. You need at least three if you want to
>>>> ; tolerate any node failures. Always create an odd number.
>>>> [mon]
>>>> mon data = /vol0/data/mon.$id
>>>>
>>>> ; some minimal logging (just message traffic) to aid debugging
>>>>
>>>> debug ms = 1 ; see message traffic
>>>> debug mon = 0 ; monitor
>>>> debug paxos = 0 ; monitor replication
>>>> debug auth = 0 ;
>>>>
>>>> mon allowed clock drift = 2
>>>>
>>>> [mon.0]
>>>> host = 10-177-64-4
>>>> mon addr = 10.177.64.4:6789
>>>>
>>>> ; radosgw client list
>>>> [client.radosgw.10-177-64-4]
>>>>
>>>> host = 10-177-64-4
>>>> log file = /var/log/ceph/$name.log
>>>> debug rgw = 1
>>>> debug ms = 1
>>>>
>>>> ; osd
>>>> ; You need at least one. Two if you want data to be replicated.
>>>> ; Define as many as you like.
>>>> [osd]
>>>> ; This is where the btrfs volume will be mounted.
>>>>
>>>> osd data = /vol0/data/osd.$id
>>>>
>>>> ; Ideally, make this a separate disk or partition. A few GB
>>>> ; is usually enough; more if you have fast disks. You can use
>>>> ; a file under the osd data dir if need be
>>>> ; (e.g. /data/osd$id/journal), but it will be slower than a
>>>> ; separate disk or partition.
>>>>
>>>> osd journal = /vol0/data/osd.$id/journal
>>>>
>>>> ; If the OSD journal is a file, you need to specify the size.
>>>> This is specified in MB.
>>>>
>>>> osd journal size = 512
>>>>
>>>> filestore journal writeahead = 1
>>>> osd heartbeat grace = 5
>>>>
>>>> debug ms = 0 ; message traffic
>>>> debug osd = 0
>>>> debug filestore = 0 ; local object storage
>>>> debug journal = 0 ; local journaling
>>>> debug monc = 0
>>>> debug rados = 1
>>>>
>>>> [osd.0]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.0
>>>> keyring = /vol0/data/osd.0/keyring
>>>>
>>>> [osd.1]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.1
>>>> keyring = /vol0/data/osd.1/keyring
>>>>
>>>> [osd.2]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.2
>>>> keyring = /vol0/data/osd.2/keyring
>>>>
>>>> [osd.3]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.3
>>>> keyring = /vol0/data/osd.3/keyring
>>>>
>>>> [osd.4]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.4
>>>> keyring = /vol0/data/osd.4/keyring
>>>>
>>>> [osd.5]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.5
>>>> keyring = /vol0/data/osd.5/keyring
>>>>
>>>> [osd.6]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.6
>>>> keyring = /vol0/data/osd.6/keyring
>>>>
>>>> [osd.7]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.7
>>>> keyring = /vol0/data/osd.7/keyring
>>>>
>>>> [osd.8]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.8
>>>> keyring = /vol0/data/osd.8/keyring
>>>>
>>>> [osd.9]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.9
>>>> keyring = /vol0/data/osd.9/keyring
>>>>
>>>> [osd.10]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.10
>>>> keyring = /vol0/data/osd.10/keyring
>>>>
>>>> [osd.11]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.11
>>>> keyring = /vol0/data/osd.11/keyring
>>>>
>>>> [osd.12]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.12
>>>> keyring = /vol0/data/osd.12/keyring
>>>>
>>>> [osd.13]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.13
>>>> keyring = /vol0/data/osd.13/keyring
>>>>
>>>> [osd.14]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.14
>>>> keyring = /vol0/data/osd.14/keyring
>>>>
>>>> [osd.15]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.15
>>>> keyring = /vol0/data/osd.15/keyring
>>>>
>>>> [osd.16]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.16
>>>> keyring = /vol0/data/osd.16/keyring
>>>>
>>>> [osd.17]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.17
>>>> keyring = /vol0/data/osd.17/keyring
>>>>
>>>> [osd.18]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.18
>>>> keyring = /vol0/data/osd.18/keyring
>>>>
>>>> [osd.19]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.19
>>>> keyring = /vol0/data/osd.19/keyring
>>>>
>>>> [osd.20]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.20
>>>> keyring = /vol0/data/osd.20/keyring
>>>>
>>>> [osd.21]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.21
>>>> keyring = /vol0/data/osd.21/keyring
>>>>
>>>> [osd.22]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.22
>>>> keyring = /vol0/data/osd.22/keyring
>>>>
>>>> [osd.23]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.23
>>>> keyring = /vol0/data/osd.23/keyring
>>>>
>>>> [osd.24]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.24
>>>> keyring = /vol0/data/osd.24/keyring
>>>>
>>>> [osd.25]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.25
>>>> keyring = /vol0/data/osd.25/keyring
>>>>
>>>>
>>>
>>
>>
>>
>
--
-----
Pozdrawiam
Sławek "sZiBis" Skowron
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Problem with attaching rbd device in qemu-kvm
2011-12-13 12:56 ` Sławomir Skowron
@ 2011-12-13 20:08 ` Josh Durgin
2011-12-16 12:17 ` Sławomir Skowron
0 siblings, 1 reply; 10+ messages in thread
From: Josh Durgin @ 2011-12-13 20:08 UTC (permalink / raw)
To: Sławomir Skowron; +Cc: ceph-devel
On 12/13/2011 04:56 AM, Sławomir Skowron wrote:
> Finally, i manage the problem with rbd with kvm 1.0, and libvirt
> 0.9.8, or i think i manage :), but i get stuck with one thing after.
>
> 2011-12-13 12:13:31.173+0000: 21512: error :
> qemuMonitorJSONCheckError:318 : internal error unable to execute QEMU
> command 'device_add': Bus 'pci.0' does not support hotplugging
> 2011-12-13 12:13:31.173+0000: 21512: warning :
> qemuDomainAttachPciDiskDevice:244 : qemuMonitorAddDevice failed on
> file=rbd:rbd/testdysk:id=admin:key=AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog==:auth_supported=cephx
> none:mon_host=10.177.64.4\:6789,if=none,id=drive-virtio-disk2,format=raw
> (virtio-blk-pci,bus=pci.0,addr=0x7,drive=drive-virtio-disk2,id=virtio-disk2)
> 2011-12-13 12:13:31.173+0000: 21512: error :
> virSecurityDACRestoreSecurityFileLabel:143 : cannot resolve symlink
> rbd/testdysk: No such file or directory2011-12-13 12:13:31.173+0000:
> 21512: warning : qemuDomainAttachPciDiskDevice:287 : Unable to restore
> security label on rbd/testdysk
> and i can't find any info about that.
>
> my kvm.xml inject file:
>
> <disk type="network" device="disk">
> <driver name="qemu" type="raw"/>
> <auth username="admin">
> <secret type="ceph"
> uuid="0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f"/>
> </auth>
> <source protocol="rbd" name="rbd/testdysk">
> <host name="10.177.64.4" port="6789"/>
> </source>
> <target dev="sdc" bus="virtio"/>
> </disk>
>
> secret:
>
> cat /tmp/secret.xml
> <secret ephemeral='no' private='no'>
> <uuid>0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f</uuid>
> <usage type='ceph'>
> <domain>rbd/admin</domain>
> <name>admin</name>
> </usage>
> </secret>
>
> virsh secret-define /tmp/secret.xml
> Secret 0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f created
>
> virsh secret-set-value 0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f
> AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog==
> Secret value set
>
Your xml and configuration are all fine, you're hitting the libvirt bug
that I mentioned before. The fix isn't in any stable release yet, and
0.9.8 was just released yesterday, so I guess you'll have to compile it
yourself or disable the libvirt security driver.
> 2011/12/12 Josh Durgin<josh.durgin@dreamhost.com>:
>> On 12/09/2011 04:48 AM, Sławomir Skowron wrote:
>>>
>>> Sorry, for my lag, but i was sick.
>>>
>>> I handled problem with apparmor before and now it's not a problem,
>>> even when i send mail before, it was solved.
>>>
>>> Ok when i create image from qemu-img it's looks like that, and works
>>> perfect, (attachment with log of this opperation.)
>>>
>>> But when i use virsh, it's still a problem. Just like he don't know
>>> what user is going to be auth.
>>>
>>> I try, as client.admin (default i hope :)) only with id=admin in
>>> kvm.xml, or client=admin, and other combinations - like client.rbd
>>> etc., Still nothing.
>>
>>
>> admin is the default, and the correct option for client.admin is id=admin,
>> or id=foo for client.foo.
>>
>>
>>>
>>> I try, improve a keyring file, adding client rbd into keyring on
>>> machine with kvm hypervisor in /etc/ceph/keyring.bin
>>>
>>> [client.rbd]
>>> key = AAAAAAAAAAAAAAAA
>>> auid = 18446744073709551615
>>> caps mds = "allow"
>>> caps mon = "allow r"
>>> caps osd = "allow rw pool=kvmtest"
>>>
>>> and on cluster site like above. I was added, a section [client.rbd]
>>> with keyring path, and it's still a problem.
>>>
>>
>> Did you run 'ceph auth add -i /etc/ceph/keyring.bin'?
>> Make sure 'ceph auth list' shows client.rbd with the right key.
>
> Overall i want to use admin id, and i think now it's correctly used in xml.
>
>>
>>
>>> 2011-12-09 13:29:11.519096 7fe616b59700 mon.0@0(leader) e1
>>> ms_verify_authorizer 10.177.32.66:0/58020608 client protocol 0
>>> root@s3-10-177-64-4:~# tail -n10000 /var/log/ceph/mon.0.log | grep
>>> 13:29:11.
>>> 2011-12-09 13:29:11.519096 7fe616b59700 mon.0@0(leader) e1
>>> ms_verify_authorizer 10.177.32.66:0/58020608 client protocol 0
>>> 2011-12-09 13:29:11.519624 7fe61b090700 -- 10.177.64.4:6789/0<==
>>> client.? 10.177.32.66:0/58020608 1 ==== auth(proto 0 21 bytes) v1 ====
>>> 47+0+0 (3310669357 0 0) 0x17f2e00 con 0x19d2c80
>>> 2011-12-09 13:29:11.519645 7fe61b090700 mon.0@0(leader) e1 have connection
>>> 2011-12-09 13:29:11.519655 7fe61b090700 mon.0@0(leader) e1 do not have
>>> session, making new one
>>> 2011-12-09 13:29:11.519668 7fe61b090700 mon.0@0(leader) e1 ms_dispatch
>>> new session MonSession: client.? 10.177.32.66:0/58020608 is open for
>>> client.? 10.177.32.66:0/58020608
>>> 2011-12-09 13:29:11.519674 7fe61b090700 mon.0@0(leader) e1 setting
>>> timeout on session
>>> 2011-12-09 13:29:11.519680 7fe61b090700 mon.0@0(leader) e1 caps
>>> 2011-12-09 13:29:11.519688 7fe61b090700 mon.0@0(leader).auth v353
>>> AuthMonitor::update_from_paxos()
>>> 2011-12-09 13:29:11.519697 7fe61b090700 mon.0@0(leader).auth v353
>>> preprocess_query auth(proto 0 21 bytes) v1 from client.?
>>> 10.177.32.66:0/58020608
>>> 2011-12-09 13:29:11.519703 7fe61b090700 mon.0@0(leader).auth v353
>>> prep_auth() blob_size=21
>>> 2011-12-09 13:29:11.519735 7fe61b090700 -- 10.177.64.4:6789/0 -->
>>> 10.177.32.66:0/58020608 -- auth_reply(proto 0 -95 Operation not
>>> supported) v1 -- ?+0 0x1246200 con 0x19d2c80
>>>
>>> from libvirt.log
>>>
>>> 23:20:56.689: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
>>> : cannot resolve symlink rbd:kvmtest/kvmtest1:name=admin: No such file
>>> or directory
>>> 23:20:56.882: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>>> Unable to restore security label on rbd:kvmtest/kvmtest1:name=admin
>>> 23:21:09.172: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>>> failed: open disk image file failed
>>> 23:21:09.172: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
>>> : cannot resolve symlink rbd:kvmtest/kvmtest1:name=admin: No such file
>>> or directory
>>> 23:21:09.364: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>>> Unable to restore security label on rbd:kvmtest/kvmtest1:name=admin
>>> 23:21:49.645: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>>> failed: open disk image file failed
>>> 23:21:49.645: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
>>> : cannot resolve symlink rbd:kvmtest/kvmtest1:id=admin: No such file
>>> or directory
>>> 23:21:49.853: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>>> Unable to restore security label on rbd:kvmtest/kvmtest1:id=admin
>>> 13:08:08.924: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>>> failed: open disk image file failed
>>> 13:08:08.924: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
>>> : cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or
>>> directory
>>> 13:08:09.117: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>>> Unable to restore security label on rbd:kvmtest/kvmtest1
>>> 13:09:45.402: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>>> failed: open disk image file failed
>>> 13:29:05.554: 1310: error : qemuMonitorTextAddDrive:2418 : operation
>>> failed: open disk image file failed
>>> 13:29:05.554: 1310: error : virSecurityDACRestoreSecurityFileLabel:143
>>> : cannot resolve symlink kvmtest/kvmtest1: No such file or directory
>>> 13:29:05.751: 1310: warning : qemuDomainAttachPciDiskDevice:250 :
>>> Unable to restore security label on kvmtest/kvmtest1
>>> 13:29:05.751: 1310: warning : virEventPollUpdateHandle:147 : Ignoring
>>> invalid update watch -1
>>
>>
>> This is libvirt mistakenly trying to treat the rbd disk as a file - fixed in
>> 20e1233c31e3150d259073c523077acdccb5d419 to libvirt. If you don't want to
>> upgrade libvirt, someone on IRC last week reported that installing
>> libapparmor1 fixed this problem for them, since libvirt no longer fell back
>> to it's own DAC implementation.
>
> hmm is this commited to any stable version now ?? i have installed
> libapparmor1,
>
> ri libapparmor1 2.7.0~beta1+bzr1774-1ubuntu2
> changehat AppArmor library
>
> and i use newest stable libvirt 0.9.8.
>
>>
>>
>>> 2011/12/1 Josh Durgin<josh.durgin@dreamhost.com>:
>>>>
>>>> On 12/01/2011 11:37 AM, Sławomir Skowron wrote:
>>>>>
>>>>>
>>>>> I have some problems. Can you help me ??
>>>>>
>>>>> ceph cluster:
>>>>> ceph 0.38, oneiric, kernel 3.0.0 x86_64 - now only one machine.
>>>>>
>>>>> kvm hypervisor:
>>>>> kvm version 1.0-rc4 (0.15.92), libvirt 0.9.2, kernel 3.0.0, Ubuntu
>>>>> oneiric x86_64.
>>>>>
>>>>> I create image from qemu-img on machine with kvm VM's, and it works very
>>>>> well:
>>>>>
>>>>> qemu-img create -f rbd rbd:kvmtest/kvmtest1 10G
>>>>> Formatting 'rbd:kvmtest/kvmtest1', fmt=rbd size=10737418240
>>>>> cluster_size=0
>>>>>
>>>>> in ceph cluster machines:
>>>>>
>>>>> rados -p kvmtest ls
>>>>> kvmtest1.rbd
>>>>> rbd_directory
>>>>> rbd_info
>>>>>
>>>>> But when i try to attach new device to kvm VM like this:
>>>>>
>>>>> virsh attach-device one-10 /tmp/kvm.xml
>>>>>
>>>>> with kvm.xml like this:
>>>>>
>>>>> <disk type='network' device='disk'>
>>>>> <driver name='qemu' type='raw' cache='writeback'/>
>>>>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
>>>>> <host name='10.177.64.4' port='6789'/>
>>>>> </source>
>>>>> <target dev='vdc' bus='virtio'/>
>>>>> </disk>
>>>>>
>>>>> output of virsh:
>>>>>
>>>>> error: Failed to attach device from /tmp/kvm.xml
>>>>> error: cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or
>>>>> directory
>>>>
>>>>
>>>>
>>>> From the monitor output below, I'm assuming this is just a bad error
>>>> message
>>>> from libvirt.
>>>>
>>>>
>>>>> and mon.0 log in ceph cluster.
>>>>>
>>>>> 2011-12-01 20:18:17.142595 7fe02dd04700 mon.0@0(leader) e1
>>>>> ms_verify_authorizer 10.177.32.66:0/25020608 client protocol 0
>>>>> 2011-12-01 20:18:17.143041 7fe03223b700 -- 10.177.64.4:6789/0<==
>>>>> client.? 10.177.32.66:0/25020608 1 ==== auth(proto 0 21 bytes) v1 ====
>>>>> 47+0+0 (3310669357 0 0) 0x29f6a00 con 0x297adc0
>>>>> 2011-12-01 20:18:17.143095 7fe03223b700 mon.0@0(leader) e1 ms_dispatch
>>>>> new session MonSession: client.? 10.177.32.66:0/25020608 is open for
>>>>> client.? 10.177.32.66:0/25020608
>>>>> 2011-12-01 20:18:17.143125 7fe03223b700 mon.0@0(leader).auth v146
>>>>> preprocess_query auth(proto 0 21 bytes) v1 from client.?
>>>>> 10.177.32.66:0/25020608
>>>>> 2011-12-01 20:18:17.143168 7fe03223b700 -- 10.177.64.4:6789/0 -->
>>>>> 10.177.32.66:0/25020608 -- auth_reply(proto 0 -95 Operation not
>>>>> supported) v1 -- ?+0 0x2f30600 con 0x297adc0
>>>>
>>>>
>>>>
>>>> Here you can see qemu isn't authenticating correctly, probably because
>>>> the
>>>> rados user isn't being set by libvirt. With your libvirt version, you'll
>>>> need to change:
>>>>
>>>>
>>>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
>>>>
>>>> to
>>>>
>>>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1:id=foo'>
>>>>
>>>> to authenticate as client.foo. Any ceph config option can be set like
>>>> that,
>>>> separated by colons.
>>>>
>>>> If this still doesn't work, I'd suggest setting
>>>> debug_rbd=20:debug_monc=20:debug_auth=20:log_to_stderr=2 to get more info
>>>> in
>>>> the libvirt instance log (usually
>>>> /var/log/libvirt/qemu/instance_name.log).
>>>> There may be apparmour restrictions preventing qemu from reading the
>>>> keyring
>>>> or ceph config files.
>>>>
>>>>
>>>>> ceph.conf, and keyring.bin exists on server, and machine with qemu-kvm
>>>>> in same directory:
>>>>> ; global
>>>>> [global]
>>>>> ; enable secure authentication
>>>>>
>>>>> auth supported = cephx
>>>>> keyring = /etc/ceph/keyring.bin
>>>>>
>>>>> debug rgw = 1
>>>>>
>>>>> rgw print continue = false
>>>>> rgw socket path = /var/run/radosgw.sock
>>>>>
>>>>> ; monitors
>>>>> ; You need at least one. You need at least three if you want to
>>>>> ; tolerate any node failures. Always create an odd number.
>>>>> [mon]
>>>>> mon data = /vol0/data/mon.$id
>>>>>
>>>>> ; some minimal logging (just message traffic) to aid debugging
>>>>>
>>>>> debug ms = 1 ; see message traffic
>>>>> debug mon = 0 ; monitor
>>>>> debug paxos = 0 ; monitor replication
>>>>> debug auth = 0 ;
>>>>>
>>>>> mon allowed clock drift = 2
>>>>>
>>>>> [mon.0]
>>>>> host = 10-177-64-4
>>>>> mon addr = 10.177.64.4:6789
>>>>>
>>>>> ; radosgw client list
>>>>> [client.radosgw.10-177-64-4]
>>>>>
>>>>> host = 10-177-64-4
>>>>> log file = /var/log/ceph/$name.log
>>>>> debug rgw = 1
>>>>> debug ms = 1
>>>>>
>>>>> ; osd
>>>>> ; You need at least one. Two if you want data to be replicated.
>>>>> ; Define as many as you like.
>>>>> [osd]
>>>>> ; This is where the btrfs volume will be mounted.
>>>>>
>>>>> osd data = /vol0/data/osd.$id
>>>>>
>>>>> ; Ideally, make this a separate disk or partition. A few GB
>>>>> ; is usually enough; more if you have fast disks. You can use
>>>>> ; a file under the osd data dir if need be
>>>>> ; (e.g. /data/osd$id/journal), but it will be slower than a
>>>>> ; separate disk or partition.
>>>>>
>>>>> osd journal = /vol0/data/osd.$id/journal
>>>>>
>>>>> ; If the OSD journal is a file, you need to specify the size.
>>>>> This is specified in MB.
>>>>>
>>>>> osd journal size = 512
>>>>>
>>>>> filestore journal writeahead = 1
>>>>> osd heartbeat grace = 5
>>>>>
>>>>> debug ms = 0 ; message traffic
>>>>> debug osd = 0
>>>>> debug filestore = 0 ; local object storage
>>>>> debug journal = 0 ; local journaling
>>>>> debug monc = 0
>>>>> debug rados = 1
>>>>>
>>>>> [osd.0]
>>>>> host = 10-177-64-4
>>>>> osd data = /vol0/data/osd.0
>>>>> keyring = /vol0/data/osd.0/keyring
>>>>>
>>>>> [osd.1]
>>>>> host = 10-177-64-4
>>>>> osd data = /vol0/data/osd.1
>>>>> keyring = /vol0/data/osd.1/keyring
>>>>>
>>>>> [osd.2]
>>>>> host = 10-177-64-4
>>>>> osd data = /vol0/data/osd.2
>>>>> keyring = /vol0/data/osd.2/keyring
>>>>>
>>>>> [osd.3]
>>>>> host = 10-177-64-4
>>>>> osd data = /vol0/data/osd.3
>>>>> keyring = /vol0/data/osd.3/keyring
>>>>>
>>>>> [osd.4]
>>>>> host = 10-177-64-4
>>>>> osd data = /vol0/data/osd.4
>>>>> keyring = /vol0/data/osd.4/keyring
>>>>>
>>>>> [osd.5]
>>>>> host = 10-177-64-4
>>>>> osd data = /vol0/data/osd.5
>>>>> keyring = /vol0/data/osd.5/keyring
>>>>>
>>>>> [osd.6]
>>>>> host = 10-177-64-4
>>>>> osd data = /vol0/data/osd.6
>>>>> keyring = /vol0/data/osd.6/keyring
>>>>>
>>>>> [osd.7]
>>>>> host = 10-177-64-4
>>>>> osd data = /vol0/data/osd.7
>>>>> keyring = /vol0/data/osd.7/keyring
>>>>>
>>>>> [osd.8]
>>>>> host = 10-177-64-4
>>>>> osd data = /vol0/data/osd.8
>>>>> keyring = /vol0/data/osd.8/keyring
>>>>>
>>>>> [osd.9]
>>>>> host = 10-177-64-4
>>>>> osd data = /vol0/data/osd.9
>>>>> keyring = /vol0/data/osd.9/keyring
>>>>>
>>>>> [osd.10]
>>>>> host = 10-177-64-4
>>>>> osd data = /vol0/data/osd.10
>>>>> keyring = /vol0/data/osd.10/keyring
>>>>>
>>>>> [osd.11]
>>>>> host = 10-177-64-4
>>>>> osd data = /vol0/data/osd.11
>>>>> keyring = /vol0/data/osd.11/keyring
>>>>>
>>>>> [osd.12]
>>>>> host = 10-177-64-4
>>>>> osd data = /vol0/data/osd.12
>>>>> keyring = /vol0/data/osd.12/keyring
>>>>>
>>>>> [osd.13]
>>>>> host = 10-177-64-4
>>>>> osd data = /vol0/data/osd.13
>>>>> keyring = /vol0/data/osd.13/keyring
>>>>>
>>>>> [osd.14]
>>>>> host = 10-177-64-4
>>>>> osd data = /vol0/data/osd.14
>>>>> keyring = /vol0/data/osd.14/keyring
>>>>>
>>>>> [osd.15]
>>>>> host = 10-177-64-4
>>>>> osd data = /vol0/data/osd.15
>>>>> keyring = /vol0/data/osd.15/keyring
>>>>>
>>>>> [osd.16]
>>>>> host = 10-177-64-4
>>>>> osd data = /vol0/data/osd.16
>>>>> keyring = /vol0/data/osd.16/keyring
>>>>>
>>>>> [osd.17]
>>>>> host = 10-177-64-4
>>>>> osd data = /vol0/data/osd.17
>>>>> keyring = /vol0/data/osd.17/keyring
>>>>>
>>>>> [osd.18]
>>>>> host = 10-177-64-4
>>>>> osd data = /vol0/data/osd.18
>>>>> keyring = /vol0/data/osd.18/keyring
>>>>>
>>>>> [osd.19]
>>>>> host = 10-177-64-4
>>>>> osd data = /vol0/data/osd.19
>>>>> keyring = /vol0/data/osd.19/keyring
>>>>>
>>>>> [osd.20]
>>>>> host = 10-177-64-4
>>>>> osd data = /vol0/data/osd.20
>>>>> keyring = /vol0/data/osd.20/keyring
>>>>>
>>>>> [osd.21]
>>>>> host = 10-177-64-4
>>>>> osd data = /vol0/data/osd.21
>>>>> keyring = /vol0/data/osd.21/keyring
>>>>>
>>>>> [osd.22]
>>>>> host = 10-177-64-4
>>>>> osd data = /vol0/data/osd.22
>>>>> keyring = /vol0/data/osd.22/keyring
>>>>>
>>>>> [osd.23]
>>>>> host = 10-177-64-4
>>>>> osd data = /vol0/data/osd.23
>>>>> keyring = /vol0/data/osd.23/keyring
>>>>>
>>>>> [osd.24]
>>>>> host = 10-177-64-4
>>>>> osd data = /vol0/data/osd.24
>>>>> keyring = /vol0/data/osd.24/keyring
>>>>>
>>>>> [osd.25]
>>>>> host = 10-177-64-4
>>>>> osd data = /vol0/data/osd.25
>>>>> keyring = /vol0/data/osd.25/keyring
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>
>
>
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Problem with attaching rbd device in qemu-kvm
2011-12-13 20:08 ` Josh Durgin
@ 2011-12-16 12:17 ` Sławomir Skowron
2011-12-19 15:37 ` Sławomir Skowron
0 siblings, 1 reply; 10+ messages in thread
From: Sławomir Skowron @ 2011-12-16 12:17 UTC (permalink / raw)
To: Josh Durgin; +Cc: ceph-devel
2011/12/13 Josh Durgin <josh.durgin@dreamhost.com>:
> On 12/13/2011 04:56 AM, Sławomir Skowron wrote:
>>
>> Finally, i manage the problem with rbd with kvm 1.0, and libvirt
>> 0.9.8, or i think i manage :), but i get stuck with one thing after.
>>
>> 2011-12-13 12:13:31.173+0000: 21512: error :
>> qemuMonitorJSONCheckError:318 : internal error unable to execute QEMU
>> command 'device_add': Bus 'pci.0' does not support hotplugging
>> 2011-12-13 12:13:31.173+0000: 21512: warning :
>> qemuDomainAttachPciDiskDevice:244 : qemuMonitorAddDevice failed on
>>
>> file=rbd:rbd/testdysk:id=admin:key=AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog==:auth_supported=cephx
>> none:mon_host=10.177.64.4\:6789,if=none,id=drive-virtio-disk2,format=raw
>>
>> (virtio-blk-pci,bus=pci.0,addr=0x7,drive=drive-virtio-disk2,id=virtio-disk2)
>> 2011-12-13 12:13:31.173+0000: 21512: error :
>> virSecurityDACRestoreSecurityFileLabel:143 : cannot resolve symlink
>> rbd/testdysk: No such file or directory2011-12-13 12:13:31.173+0000:
>> 21512: warning : qemuDomainAttachPciDiskDevice:287 : Unable to restore
>> security label on rbd/testdysk
>> and i can't find any info about that.
>>
>> my kvm.xml inject file:
>>
>> <disk type="network" device="disk">
>> <driver name="qemu" type="raw"/>
>> <auth username="admin">
>> <secret type="ceph"
>> uuid="0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f"/>
>> </auth>
>> <source protocol="rbd" name="rbd/testdysk">
>> <host name="10.177.64.4" port="6789"/>
>> </source>
>> <target dev="sdc" bus="virtio"/>
>> </disk>
>>
>> secret:
>>
>> cat /tmp/secret.xml
>> <secret ephemeral='no' private='no'>
>> <uuid>0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f</uuid>
>> <usage type='ceph'>
>> <domain>rbd/admin</domain>
>> <name>admin</name>
>> </usage>
>> </secret>
>>
>> virsh secret-define /tmp/secret.xml
>> Secret 0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f created
>>
>> virsh secret-set-value 0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f
>> AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog==
>> Secret value set
>>
>
> Your xml and configuration are all fine, you're hitting the libvirt bug that
> I mentioned before. The fix isn't in any stable release yet, and 0.9.8 was
> just released yesterday, so I guess you'll have to compile it yourself or
> disable the libvirt security driver.
Yes libvirt 0.9.8 with this patch solved problem with rbd, now it's
clean. Thanks for that.
But, a second problem appear.
error: Failed to attach device from /root/kvm-ceph.xml
error: internal error unable to execute QEMU command 'device_add':
Property 'virtio-blk-pci.drive' can't find value 'drive-virtio-disk3'
/var/log/libvirt/libvirtd.log
2011-12-16 12:03:04.950+0000: 1070: error :
qemuMonitorJSONCheckError:318 : internal error unable to execute QEMU
command 'device_add': Property 'virtio-blk-pci.drive' can't find value
'drive-virtio-disk3'
2011-12-16 12:03:04.950+0000: 1070: warning :
qemuDomainAttachPciDiskDevice:244 : qemuMonitorAddDevice failed on
file=rbd:rbd/testdysk:rbd_writeback_window=8000000:id=admin:key=AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog==:auth_supported=cephx
none:mon_host=10.177.64.4\:6789,if=none,id=drive-virtio-disk3,format=raw
(virtio-blk-pci,bus=pci.0,addr=0xd,drive=drive-virtio-disk3,id=virtio-disk3)
if i attach a scsi device from img file everything is OK.
Anyone have, any concept with this ??
>
>
>> 2011/12/12 Josh Durgin<josh.durgin@dreamhost.com>:
>>>
>>> On 12/09/2011 04:48 AM, Sławomir Skowron wrote:
>>>>
>>>>
>>>> Sorry, for my lag, but i was sick.
>>>>
>>>> I handled problem with apparmor before and now it's not a problem,
>>>> even when i send mail before, it was solved.
>>>>
>>>> Ok when i create image from qemu-img it's looks like that, and works
>>>> perfect, (attachment with log of this opperation.)
>>>>
>>>> But when i use virsh, it's still a problem. Just like he don't know
>>>> what user is going to be auth.
>>>>
>>>> I try, as client.admin (default i hope :)) only with id=admin in
>>>> kvm.xml, or client=admin, and other combinations - like client.rbd
>>>> etc., Still nothing.
>>>
>>>
>>>
>>> admin is the default, and the correct option for client.admin is
>>> id=admin,
>>> or id=foo for client.foo.
>>>
>>>
>>>>
>>>> I try, improve a keyring file, adding client rbd into keyring on
>>>> machine with kvm hypervisor in /etc/ceph/keyring.bin
>>>>
>>>> [client.rbd]
>>>> key = AAAAAAAAAAAAAAAA
>>>> auid = 18446744073709551615
>>>> caps mds = "allow"
>>>> caps mon = "allow r"
>>>> caps osd = "allow rw pool=kvmtest"
>>>>
>>>> and on cluster site like above. I was added, a section [client.rbd]
>>>> with keyring path, and it's still a problem.
>>>>
>>>
>>> Did you run 'ceph auth add -i /etc/ceph/keyring.bin'?
>>> Make sure 'ceph auth list' shows client.rbd with the right key.
>>
>>
>> Overall i want to use admin id, and i think now it's correctly used in
>> xml.
>>
>>>
>>>
>>>> 2011-12-09 13:29:11.519096 7fe616b59700 mon.0@0(leader) e1
>>>> ms_verify_authorizer 10.177.32.66:0/58020608 client protocol 0
>>>> root@s3-10-177-64-4:~# tail -n10000 /var/log/ceph/mon.0.log | grep
>>>> 13:29:11.
>>>> 2011-12-09 13:29:11.519096 7fe616b59700 mon.0@0(leader) e1
>>>> ms_verify_authorizer 10.177.32.66:0/58020608 client protocol 0
>>>> 2011-12-09 13:29:11.519624 7fe61b090700 -- 10.177.64.4:6789/0<==
>>>> client.? 10.177.32.66:0/58020608 1 ==== auth(proto 0 21 bytes) v1 ====
>>>> 47+0+0 (3310669357 0 0) 0x17f2e00 con 0x19d2c80
>>>> 2011-12-09 13:29:11.519645 7fe61b090700 mon.0@0(leader) e1 have
>>>> connection
>>>> 2011-12-09 13:29:11.519655 7fe61b090700 mon.0@0(leader) e1 do not have
>>>> session, making new one
>>>> 2011-12-09 13:29:11.519668 7fe61b090700 mon.0@0(leader) e1 ms_dispatch
>>>> new session MonSession: client.? 10.177.32.66:0/58020608 is open for
>>>> client.? 10.177.32.66:0/58020608
>>>> 2011-12-09 13:29:11.519674 7fe61b090700 mon.0@0(leader) e1 setting
>>>> timeout on session
>>>> 2011-12-09 13:29:11.519680 7fe61b090700 mon.0@0(leader) e1 caps
>>>> 2011-12-09 13:29:11.519688 7fe61b090700 mon.0@0(leader).auth v353
>>>> AuthMonitor::update_from_paxos()
>>>> 2011-12-09 13:29:11.519697 7fe61b090700 mon.0@0(leader).auth v353
>>>> preprocess_query auth(proto 0 21 bytes) v1 from client.?
>>>> 10.177.32.66:0/58020608
>>>> 2011-12-09 13:29:11.519703 7fe61b090700 mon.0@0(leader).auth v353
>>>> prep_auth() blob_size=21
>>>> 2011-12-09 13:29:11.519735 7fe61b090700 -- 10.177.64.4:6789/0 -->
>>>> 10.177.32.66:0/58020608 -- auth_reply(proto 0 -95 Operation not
>>>> supported) v1 -- ?+0 0x1246200 con 0x19d2c80
>>>>
>>>> from libvirt.log
>>>>
>>>> 23:20:56.689: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
>>>> : cannot resolve symlink rbd:kvmtest/kvmtest1:name=admin: No such file
>>>> or directory
>>>> 23:20:56.882: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>>>> Unable to restore security label on rbd:kvmtest/kvmtest1:name=admin
>>>> 23:21:09.172: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>>>> failed: open disk image file failed
>>>> 23:21:09.172: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
>>>> : cannot resolve symlink rbd:kvmtest/kvmtest1:name=admin: No such file
>>>> or directory
>>>> 23:21:09.364: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>>>> Unable to restore security label on rbd:kvmtest/kvmtest1:name=admin
>>>> 23:21:49.645: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>>>> failed: open disk image file failed
>>>> 23:21:49.645: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
>>>> : cannot resolve symlink rbd:kvmtest/kvmtest1:id=admin: No such file
>>>> or directory
>>>> 23:21:49.853: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>>>> Unable to restore security label on rbd:kvmtest/kvmtest1:id=admin
>>>> 13:08:08.924: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>>>> failed: open disk image file failed
>>>> 13:08:08.924: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
>>>> : cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or
>>>> directory
>>>> 13:08:09.117: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>>>> Unable to restore security label on rbd:kvmtest/kvmtest1
>>>> 13:09:45.402: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>>>> failed: open disk image file failed
>>>> 13:29:05.554: 1310: error : qemuMonitorTextAddDrive:2418 : operation
>>>> failed: open disk image file failed
>>>> 13:29:05.554: 1310: error : virSecurityDACRestoreSecurityFileLabel:143
>>>> : cannot resolve symlink kvmtest/kvmtest1: No such file or directory
>>>> 13:29:05.751: 1310: warning : qemuDomainAttachPciDiskDevice:250 :
>>>> Unable to restore security label on kvmtest/kvmtest1
>>>> 13:29:05.751: 1310: warning : virEventPollUpdateHandle:147 : Ignoring
>>>> invalid update watch -1
>>>
>>>
>>>
>>> This is libvirt mistakenly trying to treat the rbd disk as a file - fixed
>>> in
>>> 20e1233c31e3150d259073c523077acdccb5d419 to libvirt. If you don't want to
>>> upgrade libvirt, someone on IRC last week reported that installing
>>> libapparmor1 fixed this problem for them, since libvirt no longer fell
>>> back
>>> to it's own DAC implementation.
>>
>>
>> hmm is this commited to any stable version now ?? i have installed
>> libapparmor1,
>>
>> ri libapparmor1 2.7.0~beta1+bzr1774-1ubuntu2
>> changehat AppArmor library
>>
>> and i use newest stable libvirt 0.9.8.
>>
>>>
>>>
>>>> 2011/12/1 Josh Durgin<josh.durgin@dreamhost.com>:
>>>>>
>>>>>
>>>>> On 12/01/2011 11:37 AM, Sławomir Skowron wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> I have some problems. Can you help me ??
>>>>>>
>>>>>> ceph cluster:
>>>>>> ceph 0.38, oneiric, kernel 3.0.0 x86_64 - now only one machine.
>>>>>>
>>>>>> kvm hypervisor:
>>>>>> kvm version 1.0-rc4 (0.15.92), libvirt 0.9.2, kernel 3.0.0, Ubuntu
>>>>>> oneiric x86_64.
>>>>>>
>>>>>> I create image from qemu-img on machine with kvm VM's, and it works
>>>>>> very
>>>>>> well:
>>>>>>
>>>>>> qemu-img create -f rbd rbd:kvmtest/kvmtest1 10G
>>>>>> Formatting 'rbd:kvmtest/kvmtest1', fmt=rbd size=10737418240
>>>>>> cluster_size=0
>>>>>>
>>>>>> in ceph cluster machines:
>>>>>>
>>>>>> rados -p kvmtest ls
>>>>>> kvmtest1.rbd
>>>>>> rbd_directory
>>>>>> rbd_info
>>>>>>
>>>>>> But when i try to attach new device to kvm VM like this:
>>>>>>
>>>>>> virsh attach-device one-10 /tmp/kvm.xml
>>>>>>
>>>>>> with kvm.xml like this:
>>>>>>
>>>>>> <disk type='network' device='disk'>
>>>>>> <driver name='qemu' type='raw' cache='writeback'/>
>>>>>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
>>>>>> <host name='10.177.64.4' port='6789'/>
>>>>>> </source>
>>>>>> <target dev='vdc' bus='virtio'/>
>>>>>> </disk>
>>>>>>
>>>>>> output of virsh:
>>>>>>
>>>>>> error: Failed to attach device from /tmp/kvm.xml
>>>>>> error: cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or
>>>>>> directory
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> From the monitor output below, I'm assuming this is just a bad error
>>>>> message
>>>>> from libvirt.
>>>>>
>>>>>
>>>>>> and mon.0 log in ceph cluster.
>>>>>>
>>>>>> 2011-12-01 20:18:17.142595 7fe02dd04700 mon.0@0(leader) e1
>>>>>> ms_verify_authorizer 10.177.32.66:0/25020608 client protocol 0
>>>>>> 2011-12-01 20:18:17.143041 7fe03223b700 -- 10.177.64.4:6789/0<==
>>>>>> client.? 10.177.32.66:0/25020608 1 ==== auth(proto 0 21 bytes) v1 ====
>>>>>> 47+0+0 (3310669357 0 0) 0x29f6a00 con 0x297adc0
>>>>>> 2011-12-01 20:18:17.143095 7fe03223b700 mon.0@0(leader) e1 ms_dispatch
>>>>>> new session MonSession: client.? 10.177.32.66:0/25020608 is open for
>>>>>> client.? 10.177.32.66:0/25020608
>>>>>> 2011-12-01 20:18:17.143125 7fe03223b700 mon.0@0(leader).auth v146
>>>>>> preprocess_query auth(proto 0 21 bytes) v1 from client.?
>>>>>> 10.177.32.66:0/25020608
>>>>>> 2011-12-01 20:18:17.143168 7fe03223b700 -- 10.177.64.4:6789/0 -->
>>>>>> 10.177.32.66:0/25020608 -- auth_reply(proto 0 -95 Operation not
>>>>>> supported) v1 -- ?+0 0x2f30600 con 0x297adc0
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Here you can see qemu isn't authenticating correctly, probably because
>>>>> the
>>>>> rados user isn't being set by libvirt. With your libvirt version,
>>>>> you'll
>>>>> need to change:
>>>>>
>>>>>
>>>>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
>>>>>
>>>>> to
>>>>>
>>>>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1:id=foo'>
>>>>>
>>>>> to authenticate as client.foo. Any ceph config option can be set like
>>>>> that,
>>>>> separated by colons.
>>>>>
>>>>> If this still doesn't work, I'd suggest setting
>>>>> debug_rbd=20:debug_monc=20:debug_auth=20:log_to_stderr=2 to get more
>>>>> info
>>>>> in
>>>>> the libvirt instance log (usually
>>>>> /var/log/libvirt/qemu/instance_name.log).
>>>>> There may be apparmour restrictions preventing qemu from reading the
>>>>> keyring
>>>>> or ceph config files.
>>>>>
>>>>>
>>>>>> ceph.conf, and keyring.bin exists on server, and machine with qemu-kvm
>>>>>> in same directory:
>>>>>> ; global
>>>>>> [global]
>>>>>> ; enable secure authentication
>>>>>>
>>>>>> auth supported = cephx
>>>>>> keyring = /etc/ceph/keyring.bin
>>>>>>
>>>>>> debug rgw = 1
>>>>>>
>>>>>> rgw print continue = false
>>>>>> rgw socket path = /var/run/radosgw.sock
>>>>>>
>>>>>> ; monitors
>>>>>> ; You need at least one. You need at least three if you want to
>>>>>> ; tolerate any node failures. Always create an odd number.
>>>>>> [mon]
>>>>>> mon data = /vol0/data/mon.$id
>>>>>>
>>>>>> ; some minimal logging (just message traffic) to aid debugging
>>>>>>
>>>>>> debug ms = 1 ; see message traffic
>>>>>> debug mon = 0 ; monitor
>>>>>> debug paxos = 0 ; monitor replication
>>>>>> debug auth = 0 ;
>>>>>>
>>>>>> mon allowed clock drift = 2
>>>>>>
>>>>>> [mon.0]
>>>>>> host = 10-177-64-4
>>>>>> mon addr = 10.177.64.4:6789
>>>>>>
>>>>>> ; radosgw client list
>>>>>> [client.radosgw.10-177-64-4]
>>>>>>
>>>>>> host = 10-177-64-4
>>>>>> log file = /var/log/ceph/$name.log
>>>>>> debug rgw = 1
>>>>>> debug ms = 1
>>>>>>
>>>>>> ; osd
>>>>>> ; You need at least one. Two if you want data to be replicated.
>>>>>> ; Define as many as you like.
>>>>>> [osd]
>>>>>> ; This is where the btrfs volume will be mounted.
>>>>>>
>>>>>> osd data = /vol0/data/osd.$id
>>>>>>
>>>>>> ; Ideally, make this a separate disk or partition. A few GB
>>>>>> ; is usually enough; more if you have fast disks. You can use
>>>>>> ; a file under the osd data dir if need be
>>>>>> ; (e.g. /data/osd$id/journal), but it will be slower than a
>>>>>> ; separate disk or partition.
>>>>>>
>>>>>> osd journal = /vol0/data/osd.$id/journal
>>>>>>
>>>>>> ; If the OSD journal is a file, you need to specify the size.
>>>>>> This is specified in MB.
>>>>>>
>>>>>> osd journal size = 512
>>>>>>
>>>>>> filestore journal writeahead = 1
>>>>>> osd heartbeat grace = 5
>>>>>>
>>>>>> debug ms = 0 ; message traffic
>>>>>> debug osd = 0
>>>>>> debug filestore = 0 ; local object storage
>>>>>> debug journal = 0 ; local journaling
>>>>>> debug monc = 0
>>>>>> debug rados = 1
>>>>>>
>>>>>> [osd.0]
>>>>>> host = 10-177-64-4
>>>>>> osd data = /vol0/data/osd.0
>>>>>> keyring = /vol0/data/osd.0/keyring
>>>>>>
>>>>>> [osd.1]
>>>>>> host = 10-177-64-4
>>>>>> osd data = /vol0/data/osd.1
>>>>>> keyring = /vol0/data/osd.1/keyring
>>>>>>
>>>>>> [osd.2]
>>>>>> host = 10-177-64-4
>>>>>> osd data = /vol0/data/osd.2
>>>>>> keyring = /vol0/data/osd.2/keyring
>>>>>>
>>>>>> [osd.3]
>>>>>> host = 10-177-64-4
>>>>>> osd data = /vol0/data/osd.3
>>>>>> keyring = /vol0/data/osd.3/keyring
>>>>>>
>>>>>> [osd.4]
>>>>>> host = 10-177-64-4
>>>>>> osd data = /vol0/data/osd.4
>>>>>> keyring = /vol0/data/osd.4/keyring
>>>>>>
>>>>>> [osd.5]
>>>>>> host = 10-177-64-4
>>>>>> osd data = /vol0/data/osd.5
>>>>>> keyring = /vol0/data/osd.5/keyring
>>>>>>
>>>>>> [osd.6]
>>>>>> host = 10-177-64-4
>>>>>> osd data = /vol0/data/osd.6
>>>>>> keyring = /vol0/data/osd.6/keyring
>>>>>>
>>>>>> [osd.7]
>>>>>> host = 10-177-64-4
>>>>>> osd data = /vol0/data/osd.7
>>>>>> keyring = /vol0/data/osd.7/keyring
>>>>>>
>>>>>> [osd.8]
>>>>>> host = 10-177-64-4
>>>>>> osd data = /vol0/data/osd.8
>>>>>> keyring = /vol0/data/osd.8/keyring
>>>>>>
>>>>>> [osd.9]
>>>>>> host = 10-177-64-4
>>>>>> osd data = /vol0/data/osd.9
>>>>>> keyring = /vol0/data/osd.9/keyring
>>>>>>
>>>>>> [osd.10]
>>>>>> host = 10-177-64-4
>>>>>> osd data = /vol0/data/osd.10
>>>>>> keyring = /vol0/data/osd.10/keyring
>>>>>>
>>>>>> [osd.11]
>>>>>> host = 10-177-64-4
>>>>>> osd data = /vol0/data/osd.11
>>>>>> keyring = /vol0/data/osd.11/keyring
>>>>>>
>>>>>> [osd.12]
>>>>>> host = 10-177-64-4
>>>>>> osd data = /vol0/data/osd.12
>>>>>> keyring = /vol0/data/osd.12/keyring
>>>>>>
>>>>>> [osd.13]
>>>>>> host = 10-177-64-4
>>>>>> osd data = /vol0/data/osd.13
>>>>>> keyring = /vol0/data/osd.13/keyring
>>>>>>
>>>>>> [osd.14]
>>>>>> host = 10-177-64-4
>>>>>> osd data = /vol0/data/osd.14
>>>>>> keyring = /vol0/data/osd.14/keyring
>>>>>>
>>>>>> [osd.15]
>>>>>> host = 10-177-64-4
>>>>>> osd data = /vol0/data/osd.15
>>>>>> keyring = /vol0/data/osd.15/keyring
>>>>>>
>>>>>> [osd.16]
>>>>>> host = 10-177-64-4
>>>>>> osd data = /vol0/data/osd.16
>>>>>> keyring = /vol0/data/osd.16/keyring
>>>>>>
>>>>>> [osd.17]
>>>>>> host = 10-177-64-4
>>>>>> osd data = /vol0/data/osd.17
>>>>>> keyring = /vol0/data/osd.17/keyring
>>>>>>
>>>>>> [osd.18]
>>>>>> host = 10-177-64-4
>>>>>> osd data = /vol0/data/osd.18
>>>>>> keyring = /vol0/data/osd.18/keyring
>>>>>>
>>>>>> [osd.19]
>>>>>> host = 10-177-64-4
>>>>>> osd data = /vol0/data/osd.19
>>>>>> keyring = /vol0/data/osd.19/keyring
>>>>>>
>>>>>> [osd.20]
>>>>>> host = 10-177-64-4
>>>>>> osd data = /vol0/data/osd.20
>>>>>> keyring = /vol0/data/osd.20/keyring
>>>>>>
>>>>>> [osd.21]
>>>>>> host = 10-177-64-4
>>>>>> osd data = /vol0/data/osd.21
>>>>>> keyring = /vol0/data/osd.21/keyring
>>>>>>
>>>>>> [osd.22]
>>>>>> host = 10-177-64-4
>>>>>> osd data = /vol0/data/osd.22
>>>>>> keyring = /vol0/data/osd.22/keyring
>>>>>>
>>>>>> [osd.23]
>>>>>> host = 10-177-64-4
>>>>>> osd data = /vol0/data/osd.23
>>>>>> keyring = /vol0/data/osd.23/keyring
>>>>>>
>>>>>> [osd.24]
>>>>>> host = 10-177-64-4
>>>>>> osd data = /vol0/data/osd.24
>>>>>> keyring = /vol0/data/osd.24/keyring
>>>>>>
>>>>>> [osd.25]
>>>>>> host = 10-177-64-4
>>>>>> osd data = /vol0/data/osd.25
>>>>>> keyring = /vol0/data/osd.25/keyring
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>>
>
--
-----
Pozdrawiam
Sławek "sZiBis" Skowron
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Problem with attaching rbd device in qemu-kvm
2011-12-16 12:17 ` Sławomir Skowron
@ 2011-12-19 15:37 ` Sławomir Skowron
2011-12-20 2:41 ` Josh Durgin
0 siblings, 1 reply; 10+ messages in thread
From: Sławomir Skowron @ 2011-12-19 15:37 UTC (permalink / raw)
To: Josh Durgin; +Cc: ceph-devel
[-- Attachment #1: Type: text/plain, Size: 21380 bytes --]
Hi,
Actual setup:
ii libvirt-bin 0.9.2-4ubuntu15.1
the programs for the libvirt library
ii libvirt0 0.9.2-4ubuntu15.1
library for interfacing with different virtualization systems
ii qemu-kvm 1.0.0+dfsg+rc2-1~oneiric1
Full virtualization on i386 and amd64 hardware
I change many thing in system, and successfully attached rbd drives via libvirt:
virsh attach-device one-888 kvm-ceph.xml
Device attached successfully
A new set of disks, appears in libvirt vm xml, but not in vm machine.
After reboot, all new rbd drives appear in VM.
Is there any chance to hotadd rbd device to working VM without reboot ??
2011/12/16 Sławomir Skowron <szibis@gmail.com>:
> 2011/12/13 Josh Durgin <josh.durgin@dreamhost.com>:
>> On 12/13/2011 04:56 AM, Sławomir Skowron wrote:
>>>
>>> Finally, i manage the problem with rbd with kvm 1.0, and libvirt
>>> 0.9.8, or i think i manage :), but i get stuck with one thing after.
>>>
>>> 2011-12-13 12:13:31.173+0000: 21512: error :
>>> qemuMonitorJSONCheckError:318 : internal error unable to execute QEMU
>>> command 'device_add': Bus 'pci.0' does not support hotplugging
>>> 2011-12-13 12:13:31.173+0000: 21512: warning :
>>> qemuDomainAttachPciDiskDevice:244 : qemuMonitorAddDevice failed on
>>>
>>> file=rbd:rbd/testdysk:id=admin:key=AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog==:auth_supported=cephx
>>> none:mon_host=10.177.64.4\:6789,if=none,id=drive-virtio-disk2,format=raw
>>>
>>> (virtio-blk-pci,bus=pci.0,addr=0x7,drive=drive-virtio-disk2,id=virtio-disk2)
>>> 2011-12-13 12:13:31.173+0000: 21512: error :
>>> virSecurityDACRestoreSecurityFileLabel:143 : cannot resolve symlink
>>> rbd/testdysk: No such file or directory2011-12-13 12:13:31.173+0000:
>>> 21512: warning : qemuDomainAttachPciDiskDevice:287 : Unable to restore
>>> security label on rbd/testdysk
>>> and i can't find any info about that.
>>>
>>> my kvm.xml inject file:
>>>
>>> <disk type="network" device="disk">
>>> <driver name="qemu" type="raw"/>
>>> <auth username="admin">
>>> <secret type="ceph"
>>> uuid="0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f"/>
>>> </auth>
>>> <source protocol="rbd" name="rbd/testdysk">
>>> <host name="10.177.64.4" port="6789"/>
>>> </source>
>>> <target dev="sdc" bus="virtio"/>
>>> </disk>
>>>
>>> secret:
>>>
>>> cat /tmp/secret.xml
>>> <secret ephemeral='no' private='no'>
>>> <uuid>0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f</uuid>
>>> <usage type='ceph'>
>>> <domain>rbd/admin</domain>
>>> <name>admin</name>
>>> </usage>
>>> </secret>
>>>
>>> virsh secret-define /tmp/secret.xml
>>> Secret 0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f created
>>>
>>> virsh secret-set-value 0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f
>>> AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog==
>>> Secret value set
>>>
>>
>> Your xml and configuration are all fine, you're hitting the libvirt bug that
>> I mentioned before. The fix isn't in any stable release yet, and 0.9.8 was
>> just released yesterday, so I guess you'll have to compile it yourself or
>> disable the libvirt security driver.
>
> Yes libvirt 0.9.8 with this patch solved problem with rbd, now it's
> clean. Thanks for that.
>
> But, a second problem appear.
>
> error: Failed to attach device from /root/kvm-ceph.xml
> error: internal error unable to execute QEMU command 'device_add':
> Property 'virtio-blk-pci.drive' can't find value 'drive-virtio-disk3'
>
> /var/log/libvirt/libvirtd.log
> 2011-12-16 12:03:04.950+0000: 1070: error :
> qemuMonitorJSONCheckError:318 : internal error unable to execute QEMU
> command 'device_add': Property 'virtio-blk-pci.drive' can't find value
> 'drive-virtio-disk3'
> 2011-12-16 12:03:04.950+0000: 1070: warning :
> qemuDomainAttachPciDiskDevice:244 : qemuMonitorAddDevice failed on
> file=rbd:rbd/testdysk:rbd_writeback_window=8000000:id=admin:key=AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog==:auth_supported=cephx
> none:mon_host=10.177.64.4\:6789,if=none,id=drive-virtio-disk3,format=raw
> (virtio-blk-pci,bus=pci.0,addr=0xd,drive=drive-virtio-disk3,id=virtio-disk3)
>
> if i attach a scsi device from img file everything is OK.
>
> Anyone have, any concept with this ??
>
>>
>>
>>> 2011/12/12 Josh Durgin<josh.durgin@dreamhost.com>:
>>>>
>>>> On 12/09/2011 04:48 AM, Sławomir Skowron wrote:
>>>>>
>>>>>
>>>>> Sorry, for my lag, but i was sick.
>>>>>
>>>>> I handled problem with apparmor before and now it's not a problem,
>>>>> even when i send mail before, it was solved.
>>>>>
>>>>> Ok when i create image from qemu-img it's looks like that, and works
>>>>> perfect, (attachment with log of this opperation.)
>>>>>
>>>>> But when i use virsh, it's still a problem. Just like he don't know
>>>>> what user is going to be auth.
>>>>>
>>>>> I try, as client.admin (default i hope :)) only with id=admin in
>>>>> kvm.xml, or client=admin, and other combinations - like client.rbd
>>>>> etc., Still nothing.
>>>>
>>>>
>>>>
>>>> admin is the default, and the correct option for client.admin is
>>>> id=admin,
>>>> or id=foo for client.foo.
>>>>
>>>>
>>>>>
>>>>> I try, improve a keyring file, adding client rbd into keyring on
>>>>> machine with kvm hypervisor in /etc/ceph/keyring.bin
>>>>>
>>>>> [client.rbd]
>>>>> key = AAAAAAAAAAAAAAAA
>>>>> auid = 18446744073709551615
>>>>> caps mds = "allow"
>>>>> caps mon = "allow r"
>>>>> caps osd = "allow rw pool=kvmtest"
>>>>>
>>>>> and on cluster site like above. I was added, a section [client.rbd]
>>>>> with keyring path, and it's still a problem.
>>>>>
>>>>
>>>> Did you run 'ceph auth add -i /etc/ceph/keyring.bin'?
>>>> Make sure 'ceph auth list' shows client.rbd with the right key.
>>>
>>>
>>> Overall i want to use admin id, and i think now it's correctly used in
>>> xml.
>>>
>>>>
>>>>
>>>>> 2011-12-09 13:29:11.519096 7fe616b59700 mon.0@0(leader) e1
>>>>> ms_verify_authorizer 10.177.32.66:0/58020608 client protocol 0
>>>>> root@s3-10-177-64-4:~# tail -n10000 /var/log/ceph/mon.0.log | grep
>>>>> 13:29:11.
>>>>> 2011-12-09 13:29:11.519096 7fe616b59700 mon.0@0(leader) e1
>>>>> ms_verify_authorizer 10.177.32.66:0/58020608 client protocol 0
>>>>> 2011-12-09 13:29:11.519624 7fe61b090700 -- 10.177.64.4:6789/0<==
>>>>> client.? 10.177.32.66:0/58020608 1 ==== auth(proto 0 21 bytes) v1 ====
>>>>> 47+0+0 (3310669357 0 0) 0x17f2e00 con 0x19d2c80
>>>>> 2011-12-09 13:29:11.519645 7fe61b090700 mon.0@0(leader) e1 have
>>>>> connection
>>>>> 2011-12-09 13:29:11.519655 7fe61b090700 mon.0@0(leader) e1 do not have
>>>>> session, making new one
>>>>> 2011-12-09 13:29:11.519668 7fe61b090700 mon.0@0(leader) e1 ms_dispatch
>>>>> new session MonSession: client.? 10.177.32.66:0/58020608 is open for
>>>>> client.? 10.177.32.66:0/58020608
>>>>> 2011-12-09 13:29:11.519674 7fe61b090700 mon.0@0(leader) e1 setting
>>>>> timeout on session
>>>>> 2011-12-09 13:29:11.519680 7fe61b090700 mon.0@0(leader) e1 caps
>>>>> 2011-12-09 13:29:11.519688 7fe61b090700 mon.0@0(leader).auth v353
>>>>> AuthMonitor::update_from_paxos()
>>>>> 2011-12-09 13:29:11.519697 7fe61b090700 mon.0@0(leader).auth v353
>>>>> preprocess_query auth(proto 0 21 bytes) v1 from client.?
>>>>> 10.177.32.66:0/58020608
>>>>> 2011-12-09 13:29:11.519703 7fe61b090700 mon.0@0(leader).auth v353
>>>>> prep_auth() blob_size=21
>>>>> 2011-12-09 13:29:11.519735 7fe61b090700 -- 10.177.64.4:6789/0 -->
>>>>> 10.177.32.66:0/58020608 -- auth_reply(proto 0 -95 Operation not
>>>>> supported) v1 -- ?+0 0x1246200 con 0x19d2c80
>>>>>
>>>>> from libvirt.log
>>>>>
>>>>> 23:20:56.689: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
>>>>> : cannot resolve symlink rbd:kvmtest/kvmtest1:name=admin: No such file
>>>>> or directory
>>>>> 23:20:56.882: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>>>>> Unable to restore security label on rbd:kvmtest/kvmtest1:name=admin
>>>>> 23:21:09.172: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>>>>> failed: open disk image file failed
>>>>> 23:21:09.172: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
>>>>> : cannot resolve symlink rbd:kvmtest/kvmtest1:name=admin: No such file
>>>>> or directory
>>>>> 23:21:09.364: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>>>>> Unable to restore security label on rbd:kvmtest/kvmtest1:name=admin
>>>>> 23:21:49.645: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>>>>> failed: open disk image file failed
>>>>> 23:21:49.645: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
>>>>> : cannot resolve symlink rbd:kvmtest/kvmtest1:id=admin: No such file
>>>>> or directory
>>>>> 23:21:49.853: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>>>>> Unable to restore security label on rbd:kvmtest/kvmtest1:id=admin
>>>>> 13:08:08.924: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>>>>> failed: open disk image file failed
>>>>> 13:08:08.924: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
>>>>> : cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or
>>>>> directory
>>>>> 13:08:09.117: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>>>>> Unable to restore security label on rbd:kvmtest/kvmtest1
>>>>> 13:09:45.402: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>>>>> failed: open disk image file failed
>>>>> 13:29:05.554: 1310: error : qemuMonitorTextAddDrive:2418 : operation
>>>>> failed: open disk image file failed
>>>>> 13:29:05.554: 1310: error : virSecurityDACRestoreSecurityFileLabel:143
>>>>> : cannot resolve symlink kvmtest/kvmtest1: No such file or directory
>>>>> 13:29:05.751: 1310: warning : qemuDomainAttachPciDiskDevice:250 :
>>>>> Unable to restore security label on kvmtest/kvmtest1
>>>>> 13:29:05.751: 1310: warning : virEventPollUpdateHandle:147 : Ignoring
>>>>> invalid update watch -1
>>>>
>>>>
>>>>
>>>> This is libvirt mistakenly trying to treat the rbd disk as a file - fixed
>>>> in
>>>> 20e1233c31e3150d259073c523077acdccb5d419 to libvirt. If you don't want to
>>>> upgrade libvirt, someone on IRC last week reported that installing
>>>> libapparmor1 fixed this problem for them, since libvirt no longer fell
>>>> back
>>>> to it's own DAC implementation.
>>>
>>>
>>> hmm is this commited to any stable version now ?? i have installed
>>> libapparmor1,
>>>
>>> ri libapparmor1 2.7.0~beta1+bzr1774-1ubuntu2
>>> changehat AppArmor library
>>>
>>> and i use newest stable libvirt 0.9.8.
>>>
>>>>
>>>>
>>>>> 2011/12/1 Josh Durgin<josh.durgin@dreamhost.com>:
>>>>>>
>>>>>>
>>>>>> On 12/01/2011 11:37 AM, Sławomir Skowron wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I have some problems. Can you help me ??
>>>>>>>
>>>>>>> ceph cluster:
>>>>>>> ceph 0.38, oneiric, kernel 3.0.0 x86_64 - now only one machine.
>>>>>>>
>>>>>>> kvm hypervisor:
>>>>>>> kvm version 1.0-rc4 (0.15.92), libvirt 0.9.2, kernel 3.0.0, Ubuntu
>>>>>>> oneiric x86_64.
>>>>>>>
>>>>>>> I create image from qemu-img on machine with kvm VM's, and it works
>>>>>>> very
>>>>>>> well:
>>>>>>>
>>>>>>> qemu-img create -f rbd rbd:kvmtest/kvmtest1 10G
>>>>>>> Formatting 'rbd:kvmtest/kvmtest1', fmt=rbd size=10737418240
>>>>>>> cluster_size=0
>>>>>>>
>>>>>>> in ceph cluster machines:
>>>>>>>
>>>>>>> rados -p kvmtest ls
>>>>>>> kvmtest1.rbd
>>>>>>> rbd_directory
>>>>>>> rbd_info
>>>>>>>
>>>>>>> But when i try to attach new device to kvm VM like this:
>>>>>>>
>>>>>>> virsh attach-device one-10 /tmp/kvm.xml
>>>>>>>
>>>>>>> with kvm.xml like this:
>>>>>>>
>>>>>>> <disk type='network' device='disk'>
>>>>>>> <driver name='qemu' type='raw' cache='writeback'/>
>>>>>>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
>>>>>>> <host name='10.177.64.4' port='6789'/>
>>>>>>> </source>
>>>>>>> <target dev='vdc' bus='virtio'/>
>>>>>>> </disk>
>>>>>>>
>>>>>>> output of virsh:
>>>>>>>
>>>>>>> error: Failed to attach device from /tmp/kvm.xml
>>>>>>> error: cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or
>>>>>>> directory
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> From the monitor output below, I'm assuming this is just a bad error
>>>>>> message
>>>>>> from libvirt.
>>>>>>
>>>>>>
>>>>>>> and mon.0 log in ceph cluster.
>>>>>>>
>>>>>>> 2011-12-01 20:18:17.142595 7fe02dd04700 mon.0@0(leader) e1
>>>>>>> ms_verify_authorizer 10.177.32.66:0/25020608 client protocol 0
>>>>>>> 2011-12-01 20:18:17.143041 7fe03223b700 -- 10.177.64.4:6789/0<==
>>>>>>> client.? 10.177.32.66:0/25020608 1 ==== auth(proto 0 21 bytes) v1 ====
>>>>>>> 47+0+0 (3310669357 0 0) 0x29f6a00 con 0x297adc0
>>>>>>> 2011-12-01 20:18:17.143095 7fe03223b700 mon.0@0(leader) e1 ms_dispatch
>>>>>>> new session MonSession: client.? 10.177.32.66:0/25020608 is open for
>>>>>>> client.? 10.177.32.66:0/25020608
>>>>>>> 2011-12-01 20:18:17.143125 7fe03223b700 mon.0@0(leader).auth v146
>>>>>>> preprocess_query auth(proto 0 21 bytes) v1 from client.?
>>>>>>> 10.177.32.66:0/25020608
>>>>>>> 2011-12-01 20:18:17.143168 7fe03223b700 -- 10.177.64.4:6789/0 -->
>>>>>>> 10.177.32.66:0/25020608 -- auth_reply(proto 0 -95 Operation not
>>>>>>> supported) v1 -- ?+0 0x2f30600 con 0x297adc0
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Here you can see qemu isn't authenticating correctly, probably because
>>>>>> the
>>>>>> rados user isn't being set by libvirt. With your libvirt version,
>>>>>> you'll
>>>>>> need to change:
>>>>>>
>>>>>>
>>>>>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
>>>>>>
>>>>>> to
>>>>>>
>>>>>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1:id=foo'>
>>>>>>
>>>>>> to authenticate as client.foo. Any ceph config option can be set like
>>>>>> that,
>>>>>> separated by colons.
>>>>>>
>>>>>> If this still doesn't work, I'd suggest setting
>>>>>> debug_rbd=20:debug_monc=20:debug_auth=20:log_to_stderr=2 to get more
>>>>>> info
>>>>>> in
>>>>>> the libvirt instance log (usually
>>>>>> /var/log/libvirt/qemu/instance_name.log).
>>>>>> There may be apparmour restrictions preventing qemu from reading the
>>>>>> keyring
>>>>>> or ceph config files.
>>>>>>
>>>>>>
>>>>>>> ceph.conf, and keyring.bin exists on server, and machine with qemu-kvm
>>>>>>> in same directory:
>>>>>>> ; global
>>>>>>> [global]
>>>>>>> ; enable secure authentication
>>>>>>>
>>>>>>> auth supported = cephx
>>>>>>> keyring = /etc/ceph/keyring.bin
>>>>>>>
>>>>>>> debug rgw = 1
>>>>>>>
>>>>>>> rgw print continue = false
>>>>>>> rgw socket path = /var/run/radosgw.sock
>>>>>>>
>>>>>>> ; monitors
>>>>>>> ; You need at least one. You need at least three if you want to
>>>>>>> ; tolerate any node failures. Always create an odd number.
>>>>>>> [mon]
>>>>>>> mon data = /vol0/data/mon.$id
>>>>>>>
>>>>>>> ; some minimal logging (just message traffic) to aid debugging
>>>>>>>
>>>>>>> debug ms = 1 ; see message traffic
>>>>>>> debug mon = 0 ; monitor
>>>>>>> debug paxos = 0 ; monitor replication
>>>>>>> debug auth = 0 ;
>>>>>>>
>>>>>>> mon allowed clock drift = 2
>>>>>>>
>>>>>>> [mon.0]
>>>>>>> host = 10-177-64-4
>>>>>>> mon addr = 10.177.64.4:6789
>>>>>>>
>>>>>>> ; radosgw client list
>>>>>>> [client.radosgw.10-177-64-4]
>>>>>>>
>>>>>>> host = 10-177-64-4
>>>>>>> log file = /var/log/ceph/$name.log
>>>>>>> debug rgw = 1
>>>>>>> debug ms = 1
>>>>>>>
>>>>>>> ; osd
>>>>>>> ; You need at least one. Two if you want data to be replicated.
>>>>>>> ; Define as many as you like.
>>>>>>> [osd]
>>>>>>> ; This is where the btrfs volume will be mounted.
>>>>>>>
>>>>>>> osd data = /vol0/data/osd.$id
>>>>>>>
>>>>>>> ; Ideally, make this a separate disk or partition. A few GB
>>>>>>> ; is usually enough; more if you have fast disks. You can use
>>>>>>> ; a file under the osd data dir if need be
>>>>>>> ; (e.g. /data/osd$id/journal), but it will be slower than a
>>>>>>> ; separate disk or partition.
>>>>>>>
>>>>>>> osd journal = /vol0/data/osd.$id/journal
>>>>>>>
>>>>>>> ; If the OSD journal is a file, you need to specify the size.
>>>>>>> This is specified in MB.
>>>>>>>
>>>>>>> osd journal size = 512
>>>>>>>
>>>>>>> filestore journal writeahead = 1
>>>>>>> osd heartbeat grace = 5
>>>>>>>
>>>>>>> debug ms = 0 ; message traffic
>>>>>>> debug osd = 0
>>>>>>> debug filestore = 0 ; local object storage
>>>>>>> debug journal = 0 ; local journaling
>>>>>>> debug monc = 0
>>>>>>> debug rados = 1
>>>>>>>
>>>>>>> [osd.0]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.0
>>>>>>> keyring = /vol0/data/osd.0/keyring
>>>>>>>
>>>>>>> [osd.1]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.1
>>>>>>> keyring = /vol0/data/osd.1/keyring
>>>>>>>
>>>>>>> [osd.2]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.2
>>>>>>> keyring = /vol0/data/osd.2/keyring
>>>>>>>
>>>>>>> [osd.3]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.3
>>>>>>> keyring = /vol0/data/osd.3/keyring
>>>>>>>
>>>>>>> [osd.4]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.4
>>>>>>> keyring = /vol0/data/osd.4/keyring
>>>>>>>
>>>>>>> [osd.5]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.5
>>>>>>> keyring = /vol0/data/osd.5/keyring
>>>>>>>
>>>>>>> [osd.6]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.6
>>>>>>> keyring = /vol0/data/osd.6/keyring
>>>>>>>
>>>>>>> [osd.7]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.7
>>>>>>> keyring = /vol0/data/osd.7/keyring
>>>>>>>
>>>>>>> [osd.8]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.8
>>>>>>> keyring = /vol0/data/osd.8/keyring
>>>>>>>
>>>>>>> [osd.9]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.9
>>>>>>> keyring = /vol0/data/osd.9/keyring
>>>>>>>
>>>>>>> [osd.10]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.10
>>>>>>> keyring = /vol0/data/osd.10/keyring
>>>>>>>
>>>>>>> [osd.11]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.11
>>>>>>> keyring = /vol0/data/osd.11/keyring
>>>>>>>
>>>>>>> [osd.12]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.12
>>>>>>> keyring = /vol0/data/osd.12/keyring
>>>>>>>
>>>>>>> [osd.13]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.13
>>>>>>> keyring = /vol0/data/osd.13/keyring
>>>>>>>
>>>>>>> [osd.14]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.14
>>>>>>> keyring = /vol0/data/osd.14/keyring
>>>>>>>
>>>>>>> [osd.15]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.15
>>>>>>> keyring = /vol0/data/osd.15/keyring
>>>>>>>
>>>>>>> [osd.16]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.16
>>>>>>> keyring = /vol0/data/osd.16/keyring
>>>>>>>
>>>>>>> [osd.17]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.17
>>>>>>> keyring = /vol0/data/osd.17/keyring
>>>>>>>
>>>>>>> [osd.18]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.18
>>>>>>> keyring = /vol0/data/osd.18/keyring
>>>>>>>
>>>>>>> [osd.19]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.19
>>>>>>> keyring = /vol0/data/osd.19/keyring
>>>>>>>
>>>>>>> [osd.20]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.20
>>>>>>> keyring = /vol0/data/osd.20/keyring
>>>>>>>
>>>>>>> [osd.21]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.21
>>>>>>> keyring = /vol0/data/osd.21/keyring
>>>>>>>
>>>>>>> [osd.22]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.22
>>>>>>> keyring = /vol0/data/osd.22/keyring
>>>>>>>
>>>>>>> [osd.23]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.23
>>>>>>> keyring = /vol0/data/osd.23/keyring
>>>>>>>
>>>>>>> [osd.24]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.24
>>>>>>> keyring = /vol0/data/osd.24/keyring
>>>>>>>
>>>>>>> [osd.25]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.25
>>>>>>> keyring = /vol0/data/osd.25/keyring
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>
>
>
>
> --
> -----
> Pozdrawiam
>
> Sławek "sZiBis" Skowron
--
-----
Pozdrawiam
Sławek "sZiBis" Skowron
[-- Attachment #2: ceph_mon_log.txt --]
[-- Type: text/plain, Size: 12455 bytes --]
2011-12-19 16:16:13.434408 7f904f32b700 mon.0@0(leader) e1 ms_verify_authorizer 10.177.32.66:0/13014391 client protocol 0
2011-12-19 16:16:13.434797 7f905426c700 -- 10.177.64.4:6789/0 <== client.? 10.177.32.66:0/13014391 1 ==== auth(proto 0 30 bytes) v1 ==== 56+0+0 (625540289 0 0) 0x11fc000 con 0xcd7280
2011-12-19 16:16:13.434814 7f905426c700 mon.0@0(leader) e1 have connection
2011-12-19 16:16:13.434821 7f905426c700 mon.0@0(leader) e1 do not have session, making new one
2011-12-19 16:16:13.434833 7f905426c700 mon.0@0(leader) e1 ms_dispatch new session MonSession: client.? 10.177.32.66:0/13014391 is open for client.? 10.177.32.66:0/13014391
2011-12-19 16:16:13.434838 7f905426c700 mon.0@0(leader) e1 setting timeout on session
2011-12-19 16:16:13.434844 7f905426c700 mon.0@0(leader) e1 caps
2011-12-19 16:16:13.434852 7f905426c700 mon.0@0(leader).auth v626 update_from_paxos()
2011-12-19 16:16:13.434861 7f905426c700 mon.0@0(leader).auth v626 preprocess_query auth(proto 0 30 bytes) v1 from client.? 10.177.32.66:0/13014391
2011-12-19 16:16:13.434867 7f905426c700 mon.0@0(leader).auth v626 prep_auth() blob_size=30
2011-12-19 16:16:13.434890 7f905426c700 mon.0@0(leader).auth v626 AuthMonitor::assign_global_id m=auth(proto 0 30 bytes) v1 mon=0/1 last_allocated=6252 max_global_id=6296
2011-12-19 16:16:13.434896 7f905426c700 mon.0@0(leader).auth v626 next_global_id should be 6253
2011-12-19 16:16:13.434936 7f905426c700 cephx server client.admin: start_session server_challenge 543a42451923b4cf
2011-12-19 16:16:13.434963 7f905426c700 -- 10.177.64.4:6789/0 --> 10.177.32.66:0/13014391 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x19b4a00 con 0xcd7280
2011-12-19 16:16:13.435651 7f905426c700 -- 10.177.64.4:6789/0 <== client.? 10.177.32.66:0/13014391 2 ==== auth(proto 2 32 bytes) v1 ==== 58+0+0 (2091681013 0 0) 0xd9c200 con 0xcd7280
2011-12-19 16:16:13.435667 7f905426c700 mon.0@0(leader) e1 have connection
2011-12-19 16:16:13.435678 7f905426c700 mon.0@0(leader) e1 ms_dispatch existing session MonSession: client.? 10.177.32.66:0/13014391 is open for client.? 10.177.32.66:0/13014391
2011-12-19 16:16:13.435684 7f905426c700 mon.0@0(leader) e1 caps
2011-12-19 16:16:13.435691 7f905426c700 mon.0@0(leader).auth v626 update_from_paxos()
2011-12-19 16:16:13.435700 7f905426c700 mon.0@0(leader).auth v626 preprocess_query auth(proto 2 32 bytes) v1 from client.? 10.177.32.66:0/13014391
2011-12-19 16:16:13.435706 7f905426c700 mon.0@0(leader).auth v626 prep_auth() blob_size=32
2011-12-19 16:16:13.435712 7f905426c700 cephx server client.admin: handle_request get_auth_session_key for client.admin
2011-12-19 16:16:13.435757 7f905426c700 cephx server client.admin: checking key: req.key=e84566446375700e expected_key=e84566446375700e
2011-12-19 16:16:13.435796 7f905426c700 cephx keyserverdata: get_service_secret service auth id 46 AQBeee5OgBTXBxAAOW5Tzjrs3mIzVrZ8N+lfsg== expires 2011-12-20 00:38:06.131500
2011-12-19 16:16:13.435811 7f905426c700 cephx: build_service_ticket_reply encoding 1 tickets with secret AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog==
2011-12-19 16:16:13.435831 7f905426c700 cephx: build_service_ticket service auth secret_id 46 ticket_info.ticket.name=client.admin
2011-12-19 16:16:13.435854 7f905426c700 cephx: service_ticket_blob is 0000 : 01 2e 00 00 00 00 00 00 00 60 00 00 00 19 86 a9 : .........`......
2011-12-19 16:16:13.435863 010 : dc d6 55 ef 5b 8d d5 54 17 68 0c 82 c2 de 57 e4 : ..U.[..T.h....W.
2011-12-19 16:16:13.435871 020 : 78 21 f3 d0 71 ef a3 17 c6 f5 88 24 e1 0b f9 ea : x!..q......$....
2011-12-19 16:16:13.435878 030 : 79 c5 b2 be 36 d3 67 09 76 e1 88 47 75 da ec 4f : y...6.g.v..Gu..O
2011-12-19 16:16:13.435886 040 : ae 75 f1 b4 24 33 a2 fc f7 f9 fd 20 a2 d5 f4 5a : .u..$3..... ...Z
2011-12-19 16:16:13.435894 050 : 1c 56 03 43 9b 1b 36 af d7 1e 99 76 22 63 83 99 : .V.C..6....v"c..
2011-12-19 16:16:13.435901 060 : ac af a4 9c 09 67 09 a4 82 7e 06 c4 6b : .....g...~..k
2011-12-19 16:16:13.435905 2011-12-19 16:16:13.435912 7f905426c700 cephx keyserverdata: get_caps: name=client.admin
2011-12-19 16:16:13.435918 7f905426c700 cephx keyserverdata: get_secret: num of caps=3
2011-12-19 16:16:13.435942 7f905426c700 -- 10.177.64.4:6789/0 --> 10.177.32.66:0/13014391 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x11fc000 con 0xcd7280
2011-12-19 16:16:13.436699 7f905426c700 -- 10.177.64.4:6789/0 <== client.? 10.177.32.66:0/13014391 3 ==== auth(proto 2 165 bytes) v1 ==== 191+0+0 (1146995391 0 0) 0x11f8800 con 0xcd7280
2011-12-19 16:16:13.436715 7f905426c700 mon.0@0(leader) e1 have connection
2011-12-19 16:16:13.436727 7f905426c700 mon.0@0(leader) e1 ms_dispatch existing session MonSession: client.? 10.177.32.66:0/13014391 is openallow * for client.? 10.177.32.66:0/13014391
2011-12-19 16:16:13.436733 7f905426c700 mon.0@0(leader) e1 caps allow *
2011-12-19 16:16:13.436739 7f905426c700 mon.0@0(leader).auth v626 update_from_paxos()
2011-12-19 16:16:13.436748 7f905426c700 mon.0@0(leader).auth v626 preprocess_query auth(proto 2 165 bytes) v1 from client.? 10.177.32.66:0/13014391
2011-12-19 16:16:13.436754 7f905426c700 mon.0@0(leader).auth v626 prep_auth() blob_size=165
2011-12-19 16:16:13.436760 7f905426c700 cephx server client.admin: handle_request get_principal_session_key
2011-12-19 16:16:13.436767 7f905426c700 cephx: verify_authorizer decrypted service auth secret_id=46
2011-12-19 16:16:13.436802 7f905426c700 cephx: verify_authorizer global_id=6253
2011-12-19 16:16:13.436828 7f905426c700 cephx: verify_authorizer ok nonce 467dc1ff70cfb3a1 reply_bl.length()=36
2011-12-19 16:16:13.436836 7f905426c700 cephx server client.admin: ticket_req.keys = 5
2011-12-19 16:16:13.436841 7f905426c700 cephx server client.admin: adding key for service mon
2011-12-19 16:16:13.436855 7f905426c700 cephx keyserverdata: get_service_secret service mon id 544 AQAhP+9O0BaxKxAABySkS5uAWPm6f/lFOhKQhA== expires 2011-12-19 16:41:53.732991
2011-12-19 16:16:13.436879 7f905426c700 cephx keyserverdata: get_caps: name=client.admin
2011-12-19 16:16:13.436886 7f905426c700 cephx keyserverdata: get_secret: num of caps=3
2011-12-19 16:16:13.436893 7f905426c700 cephx server client.admin: adding key for service osd
2011-12-19 16:16:13.436905 7f905426c700 cephx keyserverdata: get_service_secret service osd id 544 AQAhP+9O4LqxKxAAwCUAcOmsSvqjkdBNjfO4sA== expires 2011-12-19 16:41:53.733047
2011-12-19 16:16:13.436922 7f905426c700 cephx keyserverdata: get_caps: name=client.admin
2011-12-19 16:16:13.436928 7f905426c700 cephx keyserverdata: get_secret: num of caps=3
2011-12-19 16:16:13.436941 7f905426c700 cephx: build_service_ticket_reply encoding 2 tickets with secret AQA9Ve9OQIn5GRAAkwGuF4Gf2EOOXCVqR1PqnA==
2011-12-19 16:16:13.436960 7f905426c700 cephx: build_service_ticket service mon secret_id 544 ticket_info.ticket.name=client.admin
2011-12-19 16:16:13.436982 7f905426c700 cephx: service_ticket_blob is 0000 : 01 20 02 00 00 00 00 00 00 70 00 00 00 1e 63 28 : . .......p....c(
2011-12-19 16:16:13.436992 010 : eb b5 cd 8e 7d 4a 33 33 d6 b0 69 31 dc 60 a8 61 : ....}J33..i1.`.a
2011-12-19 16:16:13.437000 020 : 54 23 55 17 6f 0a 86 54 f1 17 64 c9 99 5d 81 80 : T#U.o..T..d..]..
2011-12-19 16:16:13.437008 030 : 2c 85 83 dc d1 fe 44 1d 0f f0 aa c2 9e b6 fb 0b : ,.....D.........
2011-12-19 16:16:13.437015 040 : 02 52 69 3e cc b8 29 24 93 6a a1 8c 7d b8 98 28 : .Ri>..)$.j..}..(
2011-12-19 16:16:13.437023 050 : dc 20 e3 24 3a 06 c5 b4 8b a3 84 9c 54 8b 6c ff : . .$:.......T.l.
2011-12-19 16:16:13.437030 060 : 90 e2 a1 73 4f 7b 7f 30 ba 2b ca 20 89 96 57 22 : ...sO{.0.+. ..W"
2011-12-19 16:16:13.437037 070 : 87 ab ce a1 df 67 b9 60 99 97 b2 e4 05 : .....g.`.....
2011-12-19 16:16:13.437041 2011-12-19 16:16:13.437057 7f905426c700 cephx: build_service_ticket service osd secret_id 544 ticket_info.ticket.name=client.admin
2011-12-19 16:16:13.437078 7f905426c700 cephx: service_ticket_blob is 0000 : 01 20 02 00 00 00 00 00 00 70 00 00 00 1e 7d 8f : . .......p....}.
2011-12-19 16:16:13.437087 010 : 25 75 55 8a 08 da 1e a1 2c 16 62 ab ea 19 30 40 : %uU.....,.b...0@
2011-12-19 16:16:13.437095 020 : 2e a9 2d 1b 83 14 c4 21 e5 f8 5a 5d 5c f3 2c 2d : ..-....!..Z]\.,-
2011-12-19 16:16:13.437102 030 : a7 6a ea 37 c9 d1 53 db a0 02 b4 df 43 f5 e0 dd : .j.7..S.....C...
2011-12-19 16:16:13.437110 040 : 29 d6 01 cf 78 fe 5e 2f a7 f9 3d 07 ae fb 08 1a : )...x.^/..=.....
2011-12-19 16:16:13.437125 050 : 4b 38 7e da 98 81 14 07 05 d3 f3 01 c9 ca 15 6f : K8~............o
2011-12-19 16:16:13.437133 060 : bb ae bd a1 7e 76 70 0f b7 e6 42 5d 64 03 0f 6b : ....~vp...B]d..k
2011-12-19 16:16:13.437140 070 : 0b 28 7a f3 62 c3 e7 91 7f e5 33 15 5f : .(z.b.....3._
2011-12-19 16:16:13.437144 2011-12-19 16:16:13.437163 7f905426c700 -- 10.177.64.4:6789/0 --> 10.177.32.66:0/13014391 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0xd9c200 con 0xcd7280
2011-12-19 16:16:13.437750 7f905426c700 -- 10.177.64.4:6789/0 <== client.? 10.177.32.66:0/13014391 4 ==== mon_subscribe({monmap=0+}) v2 ==== 23+0+0 (1620593354 0 0) 0x128ea80 con 0xcd7280
2011-12-19 16:16:13.437768 7f905426c700 mon.0@0(leader) e1 have connection
2011-12-19 16:16:13.437786 7f905426c700 mon.0@0(leader) e1 ms_dispatch existing session MonSession: client.? 10.177.32.66:0/13014391 is openallow * for client.? 10.177.32.66:0/13014391
2011-12-19 16:16:13.437796 7f905426c700 mon.0@0(leader) e1 caps allow *
2011-12-19 16:16:13.437837 7f905426c700 mon.0@0(leader) e1 handle_subscribe mon_subscribe({monmap=0+}) v2
2011-12-19 16:16:13.437852 7f905426c700 mon.0@0(leader) e1 check_sub monmap next 0 have 1
2011-12-19 16:16:13.437875 7f905426c700 -- 10.177.64.4:6789/0 --> 10.177.32.66:0/13014391 -- mon_map v1 -- ?+0 0xea5a80 con 0xcd7280
2011-12-19 16:16:13.437899 7f905426c700 -- 10.177.64.4:6789/0 --> client.? 10.177.32.66:0/13014391 -- mon_subscribe_ack(300s) v1 -- ?+0 0xd48600
2011-12-19 16:16:13.437919 7f905426c700 -- 10.177.64.4:6789/0 <== client.6253 10.177.32.66:0/13014391 5 ==== mon_subscribe({monmap=0+,osdmap=0}) v2 ==== 42+0+0 (982583713 0 0) 0x14fd700 con 0xcd7280
2011-12-19 16:16:13.437926 7f905426c700 mon.0@0(leader) e1 have connection
2011-12-19 16:16:13.437937 7f905426c700 mon.0@0(leader) e1 ms_dispatch existing session MonSession: client.? 10.177.32.66:0/13014391 is openallow * for client.? 10.177.32.66:0/13014391
2011-12-19 16:16:13.437942 7f905426c700 mon.0@0(leader) e1 caps allow *
2011-12-19 16:16:13.437949 7f905426c700 mon.0@0(leader) e1 handle_subscribe mon_subscribe({monmap=0+,osdmap=0}) v2
2011-12-19 16:16:13.437957 7f905426c700 mon.0@0(leader) e1 check_sub monmap next 0 have 1
2011-12-19 16:16:13.437969 7f905426c700 -- 10.177.64.4:6789/0 --> 10.177.32.66:0/13014391 -- mon_map v1 -- ?+0 0x128ea80 con 0xcd7280
2011-12-19 16:16:13.438052 7f905426c700 -- 10.177.64.4:6789/0 --> client.? 10.177.32.66:0/13014391 -- osd_map(461..461 src has 1..461) v1 -- ?+0 0x11f8800
2011-12-19 16:16:13.438067 7f905426c700 -- 10.177.64.4:6789/0 --> client.6253 10.177.32.66:0/13014391 -- mon_subscribe_ack(300s) v1 -- ?+0 0x1851000
2011-12-19 16:16:13.438083 7f905426c700 -- 10.177.64.4:6789/0 <== client.6253 10.177.32.66:0/13014391 6 ==== mon_subscribe({monmap=0+,osdmap=0}) v2 ==== 42+0+0 (982583713 0 0) 0x14fd8c0 con 0xcd7280
2011-12-19 16:16:13.438091 7f905426c700 mon.0@0(leader) e1 have connection
2011-12-19 16:16:13.438100 7f905426c700 mon.0@0(leader) e1 ms_dispatch existing session MonSession: client.? 10.177.32.66:0/13014391 is openallow * for client.? 10.177.32.66:0/13014391
2011-12-19 16:16:13.438106 7f905426c700 mon.0@0(leader) e1 caps allow *
2011-12-19 16:16:13.438113 7f905426c700 mon.0@0(leader) e1 handle_subscribe mon_subscribe({monmap=0+,osdmap=0}) v2
2011-12-19 16:16:13.438120 7f905426c700 mon.0@0(leader) e1 check_sub monmap next 0 have 1
2011-12-19 16:16:13.438132 7f905426c700 -- 10.177.64.4:6789/0 --> 10.177.32.66:0/13014391 -- mon_map v1 -- ?+0 0x14fd700 con 0xcd7280
2011-12-19 16:16:13.438234 7f905426c700 -- 10.177.64.4:6789/0 --> client.? 10.177.32.66:0/13014391 -- osd_map(461..461 src has 1..461) v1 -- ?+0 0x19b4200
2011-12-19 16:16:13.438258 7f905426c700 -- 10.177.64.4:6789/0 --> client.6253 10.177.32.66:0/13014391 -- mon_subscribe_ack(300s) v1 -- ?+0 0xda7780
2011-12-19 16:16:13.649247 7f905426c700 -- 10.177.64.4:6789/0 <== osd.25 10.177.64.4:6851/5170 162 ==== pg_stats(0 pgs tid 72 v 461) v1 ==== 275+0+0 (1103829560 0 0) 0xd89d80 con 0xcd4dc0
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Problem with attaching rbd device in qemu-kvm
2011-12-19 15:37 ` Sławomir Skowron
@ 2011-12-20 2:41 ` Josh Durgin
2011-12-20 9:38 ` Sławomir Skowron
0 siblings, 1 reply; 10+ messages in thread
From: Josh Durgin @ 2011-12-20 2:41 UTC (permalink / raw)
To: Sławomir Skowron; +Cc: ceph-devel
On 12/19/2011 07:37 AM, Sławomir Skowron wrote:
> Hi,
>
> Actual setup:
>
> ii libvirt-bin 0.9.2-4ubuntu15.1
> the programs for the libvirt library
> ii libvirt0 0.9.2-4ubuntu15.1
> library for interfacing with different virtualization systems
> ii qemu-kvm 1.0.0+dfsg+rc2-1~oneiric1
> Full virtualization on i386 and amd64 hardware
>
> I change many thing in system, and successfully attached rbd drives via libvirt:
>
> virsh attach-device one-888 kvm-ceph.xml
> Device attached successfully
>
> A new set of disks, appears in libvirt vm xml, but not in vm machine.
> After reboot, all new rbd drives appear in VM.
>
> Is there any chance to hotadd rbd device to working VM without reboot ??
Hotplugging should work, but it does require guest support. Does your
guest kernel have hotplugging enabled? Does hotplugging work with a raw
image file using the same configuration?
>
> 2011/12/16 Sławomir Skowron<szibis@gmail.com>:
>> 2011/12/13 Josh Durgin<josh.durgin@dreamhost.com>:
>>> On 12/13/2011 04:56 AM, Sławomir Skowron wrote:
>>>>
>>>> Finally, i manage the problem with rbd with kvm 1.0, and libvirt
>>>> 0.9.8, or i think i manage :), but i get stuck with one thing after.
>>>>
>>>> 2011-12-13 12:13:31.173+0000: 21512: error :
>>>> qemuMonitorJSONCheckError:318 : internal error unable to execute QEMU
>>>> command 'device_add': Bus 'pci.0' does not support hotplugging
>>>> 2011-12-13 12:13:31.173+0000: 21512: warning :
>>>> qemuDomainAttachPciDiskDevice:244 : qemuMonitorAddDevice failed on
>>>>
>>>> file=rbd:rbd/testdysk:id=admin:key=AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog==:auth_supported=cephx
>>>> none:mon_host=10.177.64.4\:6789,if=none,id=drive-virtio-disk2,format=raw
>>>>
>>>> (virtio-blk-pci,bus=pci.0,addr=0x7,drive=drive-virtio-disk2,id=virtio-disk2)
>>>> 2011-12-13 12:13:31.173+0000: 21512: error :
>>>> virSecurityDACRestoreSecurityFileLabel:143 : cannot resolve symlink
>>>> rbd/testdysk: No such file or directory2011-12-13 12:13:31.173+0000:
>>>> 21512: warning : qemuDomainAttachPciDiskDevice:287 : Unable to restore
>>>> security label on rbd/testdysk
>>>> and i can't find any info about that.
>>>>
>>>> my kvm.xml inject file:
>>>>
>>>> <disk type="network" device="disk">
>>>> <driver name="qemu" type="raw"/>
>>>> <auth username="admin">
>>>> <secret type="ceph"
>>>> uuid="0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f"/>
>>>> </auth>
>>>> <source protocol="rbd" name="rbd/testdysk">
>>>> <host name="10.177.64.4" port="6789"/>
>>>> </source>
>>>> <target dev="sdc" bus="virtio"/>
>>>> </disk>
>>>>
>>>> secret:
>>>>
>>>> cat /tmp/secret.xml
>>>> <secret ephemeral='no' private='no'>
>>>> <uuid>0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f</uuid>
>>>> <usage type='ceph'>
>>>> <domain>rbd/admin</domain>
>>>> <name>admin</name>
>>>> </usage>
>>>> </secret>
>>>>
>>>> virsh secret-define /tmp/secret.xml
>>>> Secret 0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f created
>>>>
>>>> virsh secret-set-value 0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f
>>>> AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog==
>>>> Secret value set
>>>>
>>>
>>> Your xml and configuration are all fine, you're hitting the libvirt bug that
>>> I mentioned before. The fix isn't in any stable release yet, and 0.9.8 was
>>> just released yesterday, so I guess you'll have to compile it yourself or
>>> disable the libvirt security driver.
>>
>> Yes libvirt 0.9.8 with this patch solved problem with rbd, now it's
>> clean. Thanks for that.
>>
>> But, a second problem appear.
>>
>> error: Failed to attach device from /root/kvm-ceph.xml
>> error: internal error unable to execute QEMU command 'device_add':
>> Property 'virtio-blk-pci.drive' can't find value 'drive-virtio-disk3'
>>
>> /var/log/libvirt/libvirtd.log
>> 2011-12-16 12:03:04.950+0000: 1070: error :
>> qemuMonitorJSONCheckError:318 : internal error unable to execute QEMU
>> command 'device_add': Property 'virtio-blk-pci.drive' can't find value
>> 'drive-virtio-disk3'
>> 2011-12-16 12:03:04.950+0000: 1070: warning :
>> qemuDomainAttachPciDiskDevice:244 : qemuMonitorAddDevice failed on
>> file=rbd:rbd/testdysk:rbd_writeback_window=8000000:id=admin:key=AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog==:auth_supported=cephx
>> none:mon_host=10.177.64.4\:6789,if=none,id=drive-virtio-disk3,format=raw
>> (virtio-blk-pci,bus=pci.0,addr=0xd,drive=drive-virtio-disk3,id=virtio-disk3)
>>
>> if i attach a scsi device from img file everything is OK.
>>
>> Anyone have, any concept with this ??
>>
>>>
>>>
>>>> 2011/12/12 Josh Durgin<josh.durgin@dreamhost.com>:
>>>>>
>>>>> On 12/09/2011 04:48 AM, Sławomir Skowron wrote:
>>>>>>
>>>>>>
>>>>>> Sorry, for my lag, but i was sick.
>>>>>>
>>>>>> I handled problem with apparmor before and now it's not a problem,
>>>>>> even when i send mail before, it was solved.
>>>>>>
>>>>>> Ok when i create image from qemu-img it's looks like that, and works
>>>>>> perfect, (attachment with log of this opperation.)
>>>>>>
>>>>>> But when i use virsh, it's still a problem. Just like he don't know
>>>>>> what user is going to be auth.
>>>>>>
>>>>>> I try, as client.admin (default i hope :)) only with id=admin in
>>>>>> kvm.xml, or client=admin, and other combinations - like client.rbd
>>>>>> etc., Still nothing.
>>>>>
>>>>>
>>>>>
>>>>> admin is the default, and the correct option for client.admin is
>>>>> id=admin,
>>>>> or id=foo for client.foo.
>>>>>
>>>>>
>>>>>>
>>>>>> I try, improve a keyring file, adding client rbd into keyring on
>>>>>> machine with kvm hypervisor in /etc/ceph/keyring.bin
>>>>>>
>>>>>> [client.rbd]
>>>>>> key = AAAAAAAAAAAAAAAA
>>>>>> auid = 18446744073709551615
>>>>>> caps mds = "allow"
>>>>>> caps mon = "allow r"
>>>>>> caps osd = "allow rw pool=kvmtest"
>>>>>>
>>>>>> and on cluster site like above. I was added, a section [client.rbd]
>>>>>> with keyring path, and it's still a problem.
>>>>>>
>>>>>
>>>>> Did you run 'ceph auth add -i /etc/ceph/keyring.bin'?
>>>>> Make sure 'ceph auth list' shows client.rbd with the right key.
>>>>
>>>>
>>>> Overall i want to use admin id, and i think now it's correctly used in
>>>> xml.
>>>>
>>>>>
>>>>>
>>>>>> 2011-12-09 13:29:11.519096 7fe616b59700 mon.0@0(leader) e1
>>>>>> ms_verify_authorizer 10.177.32.66:0/58020608 client protocol 0
>>>>>> root@s3-10-177-64-4:~# tail -n10000 /var/log/ceph/mon.0.log | grep
>>>>>> 13:29:11.
>>>>>> 2011-12-09 13:29:11.519096 7fe616b59700 mon.0@0(leader) e1
>>>>>> ms_verify_authorizer 10.177.32.66:0/58020608 client protocol 0
>>>>>> 2011-12-09 13:29:11.519624 7fe61b090700 -- 10.177.64.4:6789/0<==
>>>>>> client.? 10.177.32.66:0/58020608 1 ==== auth(proto 0 21 bytes) v1 ====
>>>>>> 47+0+0 (3310669357 0 0) 0x17f2e00 con 0x19d2c80
>>>>>> 2011-12-09 13:29:11.519645 7fe61b090700 mon.0@0(leader) e1 have
>>>>>> connection
>>>>>> 2011-12-09 13:29:11.519655 7fe61b090700 mon.0@0(leader) e1 do not have
>>>>>> session, making new one
>>>>>> 2011-12-09 13:29:11.519668 7fe61b090700 mon.0@0(leader) e1 ms_dispatch
>>>>>> new session MonSession: client.? 10.177.32.66:0/58020608 is open for
>>>>>> client.? 10.177.32.66:0/58020608
>>>>>> 2011-12-09 13:29:11.519674 7fe61b090700 mon.0@0(leader) e1 setting
>>>>>> timeout on session
>>>>>> 2011-12-09 13:29:11.519680 7fe61b090700 mon.0@0(leader) e1 caps
>>>>>> 2011-12-09 13:29:11.519688 7fe61b090700 mon.0@0(leader).auth v353
>>>>>> AuthMonitor::update_from_paxos()
>>>>>> 2011-12-09 13:29:11.519697 7fe61b090700 mon.0@0(leader).auth v353
>>>>>> preprocess_query auth(proto 0 21 bytes) v1 from client.?
>>>>>> 10.177.32.66:0/58020608
>>>>>> 2011-12-09 13:29:11.519703 7fe61b090700 mon.0@0(leader).auth v353
>>>>>> prep_auth() blob_size=21
>>>>>> 2011-12-09 13:29:11.519735 7fe61b090700 -- 10.177.64.4:6789/0 -->
>>>>>> 10.177.32.66:0/58020608 -- auth_reply(proto 0 -95 Operation not
>>>>>> supported) v1 -- ?+0 0x1246200 con 0x19d2c80
>>>>>>
>>>>>> from libvirt.log
>>>>>>
>>>>>> 23:20:56.689: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
>>>>>> : cannot resolve symlink rbd:kvmtest/kvmtest1:name=admin: No such file
>>>>>> or directory
>>>>>> 23:20:56.882: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>>>>>> Unable to restore security label on rbd:kvmtest/kvmtest1:name=admin
>>>>>> 23:21:09.172: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>>>>>> failed: open disk image file failed
>>>>>> 23:21:09.172: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
>>>>>> : cannot resolve symlink rbd:kvmtest/kvmtest1:name=admin: No such file
>>>>>> or directory
>>>>>> 23:21:09.364: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>>>>>> Unable to restore security label on rbd:kvmtest/kvmtest1:name=admin
>>>>>> 23:21:49.645: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>>>>>> failed: open disk image file failed
>>>>>> 23:21:49.645: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
>>>>>> : cannot resolve symlink rbd:kvmtest/kvmtest1:id=admin: No such file
>>>>>> or directory
>>>>>> 23:21:49.853: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>>>>>> Unable to restore security label on rbd:kvmtest/kvmtest1:id=admin
>>>>>> 13:08:08.924: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>>>>>> failed: open disk image file failed
>>>>>> 13:08:08.924: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
>>>>>> : cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or
>>>>>> directory
>>>>>> 13:08:09.117: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>>>>>> Unable to restore security label on rbd:kvmtest/kvmtest1
>>>>>> 13:09:45.402: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>>>>>> failed: open disk image file failed
>>>>>> 13:29:05.554: 1310: error : qemuMonitorTextAddDrive:2418 : operation
>>>>>> failed: open disk image file failed
>>>>>> 13:29:05.554: 1310: error : virSecurityDACRestoreSecurityFileLabel:143
>>>>>> : cannot resolve symlink kvmtest/kvmtest1: No such file or directory
>>>>>> 13:29:05.751: 1310: warning : qemuDomainAttachPciDiskDevice:250 :
>>>>>> Unable to restore security label on kvmtest/kvmtest1
>>>>>> 13:29:05.751: 1310: warning : virEventPollUpdateHandle:147 : Ignoring
>>>>>> invalid update watch -1
>>>>>
>>>>>
>>>>>
>>>>> This is libvirt mistakenly trying to treat the rbd disk as a file - fixed
>>>>> in
>>>>> 20e1233c31e3150d259073c523077acdccb5d419 to libvirt. If you don't want to
>>>>> upgrade libvirt, someone on IRC last week reported that installing
>>>>> libapparmor1 fixed this problem for them, since libvirt no longer fell
>>>>> back
>>>>> to it's own DAC implementation.
>>>>
>>>>
>>>> hmm is this commited to any stable version now ?? i have installed
>>>> libapparmor1,
>>>>
>>>> ri libapparmor1 2.7.0~beta1+bzr1774-1ubuntu2
>>>> changehat AppArmor library
>>>>
>>>> and i use newest stable libvirt 0.9.8.
>>>>
>>>>>
>>>>>
>>>>>> 2011/12/1 Josh Durgin<josh.durgin@dreamhost.com>:
>>>>>>>
>>>>>>>
>>>>>>> On 12/01/2011 11:37 AM, Sławomir Skowron wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I have some problems. Can you help me ??
>>>>>>>>
>>>>>>>> ceph cluster:
>>>>>>>> ceph 0.38, oneiric, kernel 3.0.0 x86_64 - now only one machine.
>>>>>>>>
>>>>>>>> kvm hypervisor:
>>>>>>>> kvm version 1.0-rc4 (0.15.92), libvirt 0.9.2, kernel 3.0.0, Ubuntu
>>>>>>>> oneiric x86_64.
>>>>>>>>
>>>>>>>> I create image from qemu-img on machine with kvm VM's, and it works
>>>>>>>> very
>>>>>>>> well:
>>>>>>>>
>>>>>>>> qemu-img create -f rbd rbd:kvmtest/kvmtest1 10G
>>>>>>>> Formatting 'rbd:kvmtest/kvmtest1', fmt=rbd size=10737418240
>>>>>>>> cluster_size=0
>>>>>>>>
>>>>>>>> in ceph cluster machines:
>>>>>>>>
>>>>>>>> rados -p kvmtest ls
>>>>>>>> kvmtest1.rbd
>>>>>>>> rbd_directory
>>>>>>>> rbd_info
>>>>>>>>
>>>>>>>> But when i try to attach new device to kvm VM like this:
>>>>>>>>
>>>>>>>> virsh attach-device one-10 /tmp/kvm.xml
>>>>>>>>
>>>>>>>> with kvm.xml like this:
>>>>>>>>
>>>>>>>> <disk type='network' device='disk'>
>>>>>>>> <driver name='qemu' type='raw' cache='writeback'/>
>>>>>>>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
>>>>>>>> <host name='10.177.64.4' port='6789'/>
>>>>>>>> </source>
>>>>>>>> <target dev='vdc' bus='virtio'/>
>>>>>>>> </disk>
>>>>>>>>
>>>>>>>> output of virsh:
>>>>>>>>
>>>>>>>> error: Failed to attach device from /tmp/kvm.xml
>>>>>>>> error: cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or
>>>>>>>> directory
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> From the monitor output below, I'm assuming this is just a bad error
>>>>>>> message
>>>>>>> from libvirt.
>>>>>>>
>>>>>>>
>>>>>>>> and mon.0 log in ceph cluster.
>>>>>>>>
>>>>>>>> 2011-12-01 20:18:17.142595 7fe02dd04700 mon.0@0(leader) e1
>>>>>>>> ms_verify_authorizer 10.177.32.66:0/25020608 client protocol 0
>>>>>>>> 2011-12-01 20:18:17.143041 7fe03223b700 -- 10.177.64.4:6789/0<==
>>>>>>>> client.? 10.177.32.66:0/25020608 1 ==== auth(proto 0 21 bytes) v1 ====
>>>>>>>> 47+0+0 (3310669357 0 0) 0x29f6a00 con 0x297adc0
>>>>>>>> 2011-12-01 20:18:17.143095 7fe03223b700 mon.0@0(leader) e1 ms_dispatch
>>>>>>>> new session MonSession: client.? 10.177.32.66:0/25020608 is open for
>>>>>>>> client.? 10.177.32.66:0/25020608
>>>>>>>> 2011-12-01 20:18:17.143125 7fe03223b700 mon.0@0(leader).auth v146
>>>>>>>> preprocess_query auth(proto 0 21 bytes) v1 from client.?
>>>>>>>> 10.177.32.66:0/25020608
>>>>>>>> 2011-12-01 20:18:17.143168 7fe03223b700 -- 10.177.64.4:6789/0 -->
>>>>>>>> 10.177.32.66:0/25020608 -- auth_reply(proto 0 -95 Operation not
>>>>>>>> supported) v1 -- ?+0 0x2f30600 con 0x297adc0
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Here you can see qemu isn't authenticating correctly, probably because
>>>>>>> the
>>>>>>> rados user isn't being set by libvirt. With your libvirt version,
>>>>>>> you'll
>>>>>>> need to change:
>>>>>>>
>>>>>>>
>>>>>>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
>>>>>>>
>>>>>>> to
>>>>>>>
>>>>>>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1:id=foo'>
>>>>>>>
>>>>>>> to authenticate as client.foo. Any ceph config option can be set like
>>>>>>> that,
>>>>>>> separated by colons.
>>>>>>>
>>>>>>> If this still doesn't work, I'd suggest setting
>>>>>>> debug_rbd=20:debug_monc=20:debug_auth=20:log_to_stderr=2 to get more
>>>>>>> info
>>>>>>> in
>>>>>>> the libvirt instance log (usually
>>>>>>> /var/log/libvirt/qemu/instance_name.log).
>>>>>>> There may be apparmour restrictions preventing qemu from reading the
>>>>>>> keyring
>>>>>>> or ceph config files.
>>>>>>>
>>>>>>>
>>>>>>>> ceph.conf, and keyring.bin exists on server, and machine with qemu-kvm
>>>>>>>> in same directory:
>>>>>>>> ; global
>>>>>>>> [global]
>>>>>>>> ; enable secure authentication
>>>>>>>>
>>>>>>>> auth supported = cephx
>>>>>>>> keyring = /etc/ceph/keyring.bin
>>>>>>>>
>>>>>>>> debug rgw = 1
>>>>>>>>
>>>>>>>> rgw print continue = false
>>>>>>>> rgw socket path = /var/run/radosgw.sock
>>>>>>>>
>>>>>>>> ; monitors
>>>>>>>> ; You need at least one. You need at least three if you want to
>>>>>>>> ; tolerate any node failures. Always create an odd number.
>>>>>>>> [mon]
>>>>>>>> mon data = /vol0/data/mon.$id
>>>>>>>>
>>>>>>>> ; some minimal logging (just message traffic) to aid debugging
>>>>>>>>
>>>>>>>> debug ms = 1 ; see message traffic
>>>>>>>> debug mon = 0 ; monitor
>>>>>>>> debug paxos = 0 ; monitor replication
>>>>>>>> debug auth = 0 ;
>>>>>>>>
>>>>>>>> mon allowed clock drift = 2
>>>>>>>>
>>>>>>>> [mon.0]
>>>>>>>> host = 10-177-64-4
>>>>>>>> mon addr = 10.177.64.4:6789
>>>>>>>>
>>>>>>>> ; radosgw client list
>>>>>>>> [client.radosgw.10-177-64-4]
>>>>>>>>
>>>>>>>> host = 10-177-64-4
>>>>>>>> log file = /var/log/ceph/$name.log
>>>>>>>> debug rgw = 1
>>>>>>>> debug ms = 1
>>>>>>>>
>>>>>>>> ; osd
>>>>>>>> ; You need at least one. Two if you want data to be replicated.
>>>>>>>> ; Define as many as you like.
>>>>>>>> [osd]
>>>>>>>> ; This is where the btrfs volume will be mounted.
>>>>>>>>
>>>>>>>> osd data = /vol0/data/osd.$id
>>>>>>>>
>>>>>>>> ; Ideally, make this a separate disk or partition. A few GB
>>>>>>>> ; is usually enough; more if you have fast disks. You can use
>>>>>>>> ; a file under the osd data dir if need be
>>>>>>>> ; (e.g. /data/osd$id/journal), but it will be slower than a
>>>>>>>> ; separate disk or partition.
>>>>>>>>
>>>>>>>> osd journal = /vol0/data/osd.$id/journal
>>>>>>>>
>>>>>>>> ; If the OSD journal is a file, you need to specify the size.
>>>>>>>> This is specified in MB.
>>>>>>>>
>>>>>>>> osd journal size = 512
>>>>>>>>
>>>>>>>> filestore journal writeahead = 1
>>>>>>>> osd heartbeat grace = 5
>>>>>>>>
>>>>>>>> debug ms = 0 ; message traffic
>>>>>>>> debug osd = 0
>>>>>>>> debug filestore = 0 ; local object storage
>>>>>>>> debug journal = 0 ; local journaling
>>>>>>>> debug monc = 0
>>>>>>>> debug rados = 1
>>>>>>>>
>>>>>>>> [osd.0]
>>>>>>>> host = 10-177-64-4
>>>>>>>> osd data = /vol0/data/osd.0
>>>>>>>> keyring = /vol0/data/osd.0/keyring
>>>>>>>>
>>>>>>>> [osd.1]
>>>>>>>> host = 10-177-64-4
>>>>>>>> osd data = /vol0/data/osd.1
>>>>>>>> keyring = /vol0/data/osd.1/keyring
>>>>>>>>
>>>>>>>> [osd.2]
>>>>>>>> host = 10-177-64-4
>>>>>>>> osd data = /vol0/data/osd.2
>>>>>>>> keyring = /vol0/data/osd.2/keyring
>>>>>>>>
>>>>>>>> [osd.3]
>>>>>>>> host = 10-177-64-4
>>>>>>>> osd data = /vol0/data/osd.3
>>>>>>>> keyring = /vol0/data/osd.3/keyring
>>>>>>>>
>>>>>>>> [osd.4]
>>>>>>>> host = 10-177-64-4
>>>>>>>> osd data = /vol0/data/osd.4
>>>>>>>> keyring = /vol0/data/osd.4/keyring
>>>>>>>>
>>>>>>>> [osd.5]
>>>>>>>> host = 10-177-64-4
>>>>>>>> osd data = /vol0/data/osd.5
>>>>>>>> keyring = /vol0/data/osd.5/keyring
>>>>>>>>
>>>>>>>> [osd.6]
>>>>>>>> host = 10-177-64-4
>>>>>>>> osd data = /vol0/data/osd.6
>>>>>>>> keyring = /vol0/data/osd.6/keyring
>>>>>>>>
>>>>>>>> [osd.7]
>>>>>>>> host = 10-177-64-4
>>>>>>>> osd data = /vol0/data/osd.7
>>>>>>>> keyring = /vol0/data/osd.7/keyring
>>>>>>>>
>>>>>>>> [osd.8]
>>>>>>>> host = 10-177-64-4
>>>>>>>> osd data = /vol0/data/osd.8
>>>>>>>> keyring = /vol0/data/osd.8/keyring
>>>>>>>>
>>>>>>>> [osd.9]
>>>>>>>> host = 10-177-64-4
>>>>>>>> osd data = /vol0/data/osd.9
>>>>>>>> keyring = /vol0/data/osd.9/keyring
>>>>>>>>
>>>>>>>> [osd.10]
>>>>>>>> host = 10-177-64-4
>>>>>>>> osd data = /vol0/data/osd.10
>>>>>>>> keyring = /vol0/data/osd.10/keyring
>>>>>>>>
>>>>>>>> [osd.11]
>>>>>>>> host = 10-177-64-4
>>>>>>>> osd data = /vol0/data/osd.11
>>>>>>>> keyring = /vol0/data/osd.11/keyring
>>>>>>>>
>>>>>>>> [osd.12]
>>>>>>>> host = 10-177-64-4
>>>>>>>> osd data = /vol0/data/osd.12
>>>>>>>> keyring = /vol0/data/osd.12/keyring
>>>>>>>>
>>>>>>>> [osd.13]
>>>>>>>> host = 10-177-64-4
>>>>>>>> osd data = /vol0/data/osd.13
>>>>>>>> keyring = /vol0/data/osd.13/keyring
>>>>>>>>
>>>>>>>> [osd.14]
>>>>>>>> host = 10-177-64-4
>>>>>>>> osd data = /vol0/data/osd.14
>>>>>>>> keyring = /vol0/data/osd.14/keyring
>>>>>>>>
>>>>>>>> [osd.15]
>>>>>>>> host = 10-177-64-4
>>>>>>>> osd data = /vol0/data/osd.15
>>>>>>>> keyring = /vol0/data/osd.15/keyring
>>>>>>>>
>>>>>>>> [osd.16]
>>>>>>>> host = 10-177-64-4
>>>>>>>> osd data = /vol0/data/osd.16
>>>>>>>> keyring = /vol0/data/osd.16/keyring
>>>>>>>>
>>>>>>>> [osd.17]
>>>>>>>> host = 10-177-64-4
>>>>>>>> osd data = /vol0/data/osd.17
>>>>>>>> keyring = /vol0/data/osd.17/keyring
>>>>>>>>
>>>>>>>> [osd.18]
>>>>>>>> host = 10-177-64-4
>>>>>>>> osd data = /vol0/data/osd.18
>>>>>>>> keyring = /vol0/data/osd.18/keyring
>>>>>>>>
>>>>>>>> [osd.19]
>>>>>>>> host = 10-177-64-4
>>>>>>>> osd data = /vol0/data/osd.19
>>>>>>>> keyring = /vol0/data/osd.19/keyring
>>>>>>>>
>>>>>>>> [osd.20]
>>>>>>>> host = 10-177-64-4
>>>>>>>> osd data = /vol0/data/osd.20
>>>>>>>> keyring = /vol0/data/osd.20/keyring
>>>>>>>>
>>>>>>>> [osd.21]
>>>>>>>> host = 10-177-64-4
>>>>>>>> osd data = /vol0/data/osd.21
>>>>>>>> keyring = /vol0/data/osd.21/keyring
>>>>>>>>
>>>>>>>> [osd.22]
>>>>>>>> host = 10-177-64-4
>>>>>>>> osd data = /vol0/data/osd.22
>>>>>>>> keyring = /vol0/data/osd.22/keyring
>>>>>>>>
>>>>>>>> [osd.23]
>>>>>>>> host = 10-177-64-4
>>>>>>>> osd data = /vol0/data/osd.23
>>>>>>>> keyring = /vol0/data/osd.23/keyring
>>>>>>>>
>>>>>>>> [osd.24]
>>>>>>>> host = 10-177-64-4
>>>>>>>> osd data = /vol0/data/osd.24
>>>>>>>> keyring = /vol0/data/osd.24/keyring
>>>>>>>>
>>>>>>>> [osd.25]
>>>>>>>> host = 10-177-64-4
>>>>>>>> osd data = /vol0/data/osd.25
>>>>>>>> keyring = /vol0/data/osd.25/keyring
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>>
>> --
>> -----
>> Pozdrawiam
>>
>> Sławek "sZiBis" Skowron
>
>
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Problem with attaching rbd device in qemu-kvm
2011-12-20 2:41 ` Josh Durgin
@ 2011-12-20 9:38 ` Sławomir Skowron
0 siblings, 0 replies; 10+ messages in thread
From: Sławomir Skowron @ 2011-12-20 9:38 UTC (permalink / raw)
To: Josh Durgin; +Cc: ceph-devel
Ehhh too long with this :) I forget to load acpiphp.
Thanks for everything, now its working beautifully. After 10h of
iozone on rbd devices, and no hang, or problem.
2011/12/20 Josh Durgin <josh.durgin@dreamhost.com>:
> On 12/19/2011 07:37 AM, Sławomir Skowron wrote:
>>
>> Hi,
>>
>> Actual setup:
>>
>> ii libvirt-bin 0.9.2-4ubuntu15.1
>> the programs for the libvirt library
>> ii libvirt0 0.9.2-4ubuntu15.1
>> library for interfacing with different virtualization systems
>> ii qemu-kvm 1.0.0+dfsg+rc2-1~oneiric1
>> Full virtualization on i386 and amd64 hardware
>>
>> I change many thing in system, and successfully attached rbd drives via
>> libvirt:
>>
>> virsh attach-device one-888 kvm-ceph.xml
>> Device attached successfully
>>
>> A new set of disks, appears in libvirt vm xml, but not in vm machine.
>> After reboot, all new rbd drives appear in VM.
>>
>> Is there any chance to hotadd rbd device to working VM without reboot ??
>
>
> Hotplugging should work, but it does require guest support. Does your guest
> kernel have hotplugging enabled? Does hotplugging work with a raw image file
> using the same configuration?
>
>
>>
>> 2011/12/16 Sławomir Skowron<szibis@gmail.com>:
>>>
>>> 2011/12/13 Josh Durgin<josh.durgin@dreamhost.com>:
>>>>
>>>> On 12/13/2011 04:56 AM, Sławomir Skowron wrote:
>>>>>
>>>>>
>>>>> Finally, i manage the problem with rbd with kvm 1.0, and libvirt
>>>>> 0.9.8, or i think i manage :), but i get stuck with one thing after.
>>>>>
>>>>> 2011-12-13 12:13:31.173+0000: 21512: error :
>>>>> qemuMonitorJSONCheckError:318 : internal error unable to execute QEMU
>>>>> command 'device_add': Bus 'pci.0' does not support hotplugging
>>>>> 2011-12-13 12:13:31.173+0000: 21512: warning :
>>>>> qemuDomainAttachPciDiskDevice:244 : qemuMonitorAddDevice failed on
>>>>>
>>>>>
>>>>> file=rbd:rbd/testdysk:id=admin:key=AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog==:auth_supported=cephx
>>>>>
>>>>> none:mon_host=10.177.64.4\:6789,if=none,id=drive-virtio-disk2,format=raw
>>>>>
>>>>>
>>>>> (virtio-blk-pci,bus=pci.0,addr=0x7,drive=drive-virtio-disk2,id=virtio-disk2)
>>>>> 2011-12-13 12:13:31.173+0000: 21512: error :
>>>>> virSecurityDACRestoreSecurityFileLabel:143 : cannot resolve symlink
>>>>> rbd/testdysk: No such file or directory2011-12-13 12:13:31.173+0000:
>>>>> 21512: warning : qemuDomainAttachPciDiskDevice:287 : Unable to restore
>>>>> security label on rbd/testdysk
>>>>> and i can't find any info about that.
>>>>>
>>>>> my kvm.xml inject file:
>>>>>
>>>>> <disk type="network" device="disk">
>>>>> <driver name="qemu" type="raw"/>
>>>>> <auth username="admin">
>>>>> <secret type="ceph"
>>>>> uuid="0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f"/>
>>>>> </auth>
>>>>> <source protocol="rbd" name="rbd/testdysk">
>>>>> <host name="10.177.64.4" port="6789"/>
>>>>> </source>
>>>>> <target dev="sdc" bus="virtio"/>
>>>>> </disk>
>>>>>
>>>>> secret:
>>>>>
>>>>> cat /tmp/secret.xml
>>>>> <secret ephemeral='no' private='no'>
>>>>> <uuid>0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f</uuid>
>>>>> <usage type='ceph'>
>>>>> <domain>rbd/admin</domain>
>>>>> <name>admin</name>
>>>>> </usage>
>>>>> </secret>
>>>>>
>>>>> virsh secret-define /tmp/secret.xml
>>>>> Secret 0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f created
>>>>>
>>>>> virsh secret-set-value 0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f
>>>>> AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog==
>>>>> Secret value set
>>>>>
>>>>
>>>> Your xml and configuration are all fine, you're hitting the libvirt bug
>>>> that
>>>> I mentioned before. The fix isn't in any stable release yet, and 0.9.8
>>>> was
>>>> just released yesterday, so I guess you'll have to compile it yourself
>>>> or
>>>> disable the libvirt security driver.
>>>
>>>
>>> Yes libvirt 0.9.8 with this patch solved problem with rbd, now it's
>>> clean. Thanks for that.
>>>
>>> But, a second problem appear.
>>>
>>> error: Failed to attach device from /root/kvm-ceph.xml
>>> error: internal error unable to execute QEMU command 'device_add':
>>> Property 'virtio-blk-pci.drive' can't find value 'drive-virtio-disk3'
>>>
>>> /var/log/libvirt/libvirtd.log
>>> 2011-12-16 12:03:04.950+0000: 1070: error :
>>> qemuMonitorJSONCheckError:318 : internal error unable to execute QEMU
>>> command 'device_add': Property 'virtio-blk-pci.drive' can't find value
>>> 'drive-virtio-disk3'
>>> 2011-12-16 12:03:04.950+0000: 1070: warning :
>>> qemuDomainAttachPciDiskDevice:244 : qemuMonitorAddDevice failed on
>>>
>>> file=rbd:rbd/testdysk:rbd_writeback_window=8000000:id=admin:key=AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog==:auth_supported=cephx
>>> none:mon_host=10.177.64.4\:6789,if=none,id=drive-virtio-disk3,format=raw
>>>
>>> (virtio-blk-pci,bus=pci.0,addr=0xd,drive=drive-virtio-disk3,id=virtio-disk3)
>>>
>>> if i attach a scsi device from img file everything is OK.
>>>
>>> Anyone have, any concept with this ??
>>>
>>>>
>>>>
>>>>> 2011/12/12 Josh Durgin<josh.durgin@dreamhost.com>:
>>>>>>
>>>>>>
>>>>>> On 12/09/2011 04:48 AM, Sławomir Skowron wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Sorry, for my lag, but i was sick.
>>>>>>>
>>>>>>> I handled problem with apparmor before and now it's not a problem,
>>>>>>> even when i send mail before, it was solved.
>>>>>>>
>>>>>>> Ok when i create image from qemu-img it's looks like that, and works
>>>>>>> perfect, (attachment with log of this opperation.)
>>>>>>>
>>>>>>> But when i use virsh, it's still a problem. Just like he don't know
>>>>>>> what user is going to be auth.
>>>>>>>
>>>>>>> I try, as client.admin (default i hope :)) only with id=admin in
>>>>>>> kvm.xml, or client=admin, and other combinations - like client.rbd
>>>>>>> etc., Still nothing.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> admin is the default, and the correct option for client.admin is
>>>>>> id=admin,
>>>>>> or id=foo for client.foo.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> I try, improve a keyring file, adding client rbd into keyring on
>>>>>>> machine with kvm hypervisor in /etc/ceph/keyring.bin
>>>>>>>
>>>>>>> [client.rbd]
>>>>>>> key = AAAAAAAAAAAAAAAA
>>>>>>> auid = 18446744073709551615
>>>>>>> caps mds = "allow"
>>>>>>> caps mon = "allow r"
>>>>>>> caps osd = "allow rw pool=kvmtest"
>>>>>>>
>>>>>>> and on cluster site like above. I was added, a section [client.rbd]
>>>>>>> with keyring path, and it's still a problem.
>>>>>>>
>>>>>>
>>>>>> Did you run 'ceph auth add -i /etc/ceph/keyring.bin'?
>>>>>> Make sure 'ceph auth list' shows client.rbd with the right key.
>>>>>
>>>>>
>>>>>
>>>>> Overall i want to use admin id, and i think now it's correctly used in
>>>>> xml.
>>>>>
>>>>>>
>>>>>>
>>>>>>> 2011-12-09 13:29:11.519096 7fe616b59700 mon.0@0(leader) e1
>>>>>>> ms_verify_authorizer 10.177.32.66:0/58020608 client protocol 0
>>>>>>> root@s3-10-177-64-4:~# tail -n10000 /var/log/ceph/mon.0.log | grep
>>>>>>> 13:29:11.
>>>>>>> 2011-12-09 13:29:11.519096 7fe616b59700 mon.0@0(leader) e1
>>>>>>> ms_verify_authorizer 10.177.32.66:0/58020608 client protocol 0
>>>>>>> 2011-12-09 13:29:11.519624 7fe61b090700 -- 10.177.64.4:6789/0<==
>>>>>>> client.? 10.177.32.66:0/58020608 1 ==== auth(proto 0 21 bytes) v1
>>>>>>> ====
>>>>>>> 47+0+0 (3310669357 0 0) 0x17f2e00 con 0x19d2c80
>>>>>>> 2011-12-09 13:29:11.519645 7fe61b090700 mon.0@0(leader) e1 have
>>>>>>> connection
>>>>>>> 2011-12-09 13:29:11.519655 7fe61b090700 mon.0@0(leader) e1 do not
>>>>>>> have
>>>>>>> session, making new one
>>>>>>> 2011-12-09 13:29:11.519668 7fe61b090700 mon.0@0(leader) e1
>>>>>>> ms_dispatch
>>>>>>> new session MonSession: client.? 10.177.32.66:0/58020608 is open for
>>>>>>> client.? 10.177.32.66:0/58020608
>>>>>>> 2011-12-09 13:29:11.519674 7fe61b090700 mon.0@0(leader) e1 setting
>>>>>>> timeout on session
>>>>>>> 2011-12-09 13:29:11.519680 7fe61b090700 mon.0@0(leader) e1 caps
>>>>>>> 2011-12-09 13:29:11.519688 7fe61b090700 mon.0@0(leader).auth v353
>>>>>>> AuthMonitor::update_from_paxos()
>>>>>>> 2011-12-09 13:29:11.519697 7fe61b090700 mon.0@0(leader).auth v353
>>>>>>> preprocess_query auth(proto 0 21 bytes) v1 from client.?
>>>>>>> 10.177.32.66:0/58020608
>>>>>>> 2011-12-09 13:29:11.519703 7fe61b090700 mon.0@0(leader).auth v353
>>>>>>> prep_auth() blob_size=21
>>>>>>> 2011-12-09 13:29:11.519735 7fe61b090700 -- 10.177.64.4:6789/0 -->
>>>>>>> 10.177.32.66:0/58020608 -- auth_reply(proto 0 -95 Operation not
>>>>>>> supported) v1 -- ?+0 0x1246200 con 0x19d2c80
>>>>>>>
>>>>>>> from libvirt.log
>>>>>>>
>>>>>>> 23:20:56.689: 1309: error :
>>>>>>> virSecurityDACRestoreSecurityFileLabel:143
>>>>>>> : cannot resolve symlink rbd:kvmtest/kvmtest1:name=admin: No such
>>>>>>> file
>>>>>>> or directory
>>>>>>> 23:20:56.882: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>>>>>>> Unable to restore security label on rbd:kvmtest/kvmtest1:name=admin
>>>>>>> 23:21:09.172: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>>>>>>> failed: open disk image file failed
>>>>>>> 23:21:09.172: 1309: error :
>>>>>>> virSecurityDACRestoreSecurityFileLabel:143
>>>>>>> : cannot resolve symlink rbd:kvmtest/kvmtest1:name=admin: No such
>>>>>>> file
>>>>>>> or directory
>>>>>>> 23:21:09.364: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>>>>>>> Unable to restore security label on rbd:kvmtest/kvmtest1:name=admin
>>>>>>> 23:21:49.645: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>>>>>>> failed: open disk image file failed
>>>>>>> 23:21:49.645: 1309: error :
>>>>>>> virSecurityDACRestoreSecurityFileLabel:143
>>>>>>> : cannot resolve symlink rbd:kvmtest/kvmtest1:id=admin: No such file
>>>>>>> or directory
>>>>>>> 23:21:49.853: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>>>>>>> Unable to restore security label on rbd:kvmtest/kvmtest1:id=admin
>>>>>>> 13:08:08.924: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>>>>>>> failed: open disk image file failed
>>>>>>> 13:08:08.924: 1309: error :
>>>>>>> virSecurityDACRestoreSecurityFileLabel:143
>>>>>>> : cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or
>>>>>>> directory
>>>>>>> 13:08:09.117: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>>>>>>> Unable to restore security label on rbd:kvmtest/kvmtest1
>>>>>>> 13:09:45.402: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>>>>>>> failed: open disk image file failed
>>>>>>> 13:29:05.554: 1310: error : qemuMonitorTextAddDrive:2418 : operation
>>>>>>> failed: open disk image file failed
>>>>>>> 13:29:05.554: 1310: error :
>>>>>>> virSecurityDACRestoreSecurityFileLabel:143
>>>>>>> : cannot resolve symlink kvmtest/kvmtest1: No such file or directory
>>>>>>> 13:29:05.751: 1310: warning : qemuDomainAttachPciDiskDevice:250 :
>>>>>>> Unable to restore security label on kvmtest/kvmtest1
>>>>>>> 13:29:05.751: 1310: warning : virEventPollUpdateHandle:147 : Ignoring
>>>>>>> invalid update watch -1
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> This is libvirt mistakenly trying to treat the rbd disk as a file -
>>>>>> fixed
>>>>>> in
>>>>>> 20e1233c31e3150d259073c523077acdccb5d419 to libvirt. If you don't want
>>>>>> to
>>>>>> upgrade libvirt, someone on IRC last week reported that installing
>>>>>> libapparmor1 fixed this problem for them, since libvirt no longer fell
>>>>>> back
>>>>>> to it's own DAC implementation.
>>>>>
>>>>>
>>>>>
>>>>> hmm is this commited to any stable version now ?? i have installed
>>>>> libapparmor1,
>>>>>
>>>>> ri libapparmor1 2.7.0~beta1+bzr1774-1ubuntu2
>>>>> changehat AppArmor library
>>>>>
>>>>> and i use newest stable libvirt 0.9.8.
>>>>>
>>>>>>
>>>>>>
>>>>>>> 2011/12/1 Josh Durgin<josh.durgin@dreamhost.com>:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 12/01/2011 11:37 AM, Sławomir Skowron wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I have some problems. Can you help me ??
>>>>>>>>>
>>>>>>>>> ceph cluster:
>>>>>>>>> ceph 0.38, oneiric, kernel 3.0.0 x86_64 - now only one machine.
>>>>>>>>>
>>>>>>>>> kvm hypervisor:
>>>>>>>>> kvm version 1.0-rc4 (0.15.92), libvirt 0.9.2, kernel 3.0.0, Ubuntu
>>>>>>>>> oneiric x86_64.
>>>>>>>>>
>>>>>>>>> I create image from qemu-img on machine with kvm VM's, and it works
>>>>>>>>> very
>>>>>>>>> well:
>>>>>>>>>
>>>>>>>>> qemu-img create -f rbd rbd:kvmtest/kvmtest1 10G
>>>>>>>>> Formatting 'rbd:kvmtest/kvmtest1', fmt=rbd size=10737418240
>>>>>>>>> cluster_size=0
>>>>>>>>>
>>>>>>>>> in ceph cluster machines:
>>>>>>>>>
>>>>>>>>> rados -p kvmtest ls
>>>>>>>>> kvmtest1.rbd
>>>>>>>>> rbd_directory
>>>>>>>>> rbd_info
>>>>>>>>>
>>>>>>>>> But when i try to attach new device to kvm VM like this:
>>>>>>>>>
>>>>>>>>> virsh attach-device one-10 /tmp/kvm.xml
>>>>>>>>>
>>>>>>>>> with kvm.xml like this:
>>>>>>>>>
>>>>>>>>> <disk type='network' device='disk'>
>>>>>>>>> <driver name='qemu' type='raw' cache='writeback'/>
>>>>>>>>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
>>>>>>>>> <host name='10.177.64.4' port='6789'/>
>>>>>>>>> </source>
>>>>>>>>> <target dev='vdc' bus='virtio'/>
>>>>>>>>> </disk>
>>>>>>>>>
>>>>>>>>> output of virsh:
>>>>>>>>>
>>>>>>>>> error: Failed to attach device from /tmp/kvm.xml
>>>>>>>>> error: cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or
>>>>>>>>> directory
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> From the monitor output below, I'm assuming this is just a bad
>>>>>>>> error
>>>>>>>> message
>>>>>>>> from libvirt.
>>>>>>>>
>>>>>>>>
>>>>>>>>> and mon.0 log in ceph cluster.
>>>>>>>>>
>>>>>>>>> 2011-12-01 20:18:17.142595 7fe02dd04700 mon.0@0(leader) e1
>>>>>>>>> ms_verify_authorizer 10.177.32.66:0/25020608 client protocol 0
>>>>>>>>> 2011-12-01 20:18:17.143041 7fe03223b700 -- 10.177.64.4:6789/0<==
>>>>>>>>> client.? 10.177.32.66:0/25020608 1 ==== auth(proto 0 21 bytes) v1
>>>>>>>>> ====
>>>>>>>>> 47+0+0 (3310669357 0 0) 0x29f6a00 con 0x297adc0
>>>>>>>>> 2011-12-01 20:18:17.143095 7fe03223b700 mon.0@0(leader) e1
>>>>>>>>> ms_dispatch
>>>>>>>>> new session MonSession: client.? 10.177.32.66:0/25020608 is open
>>>>>>>>> for
>>>>>>>>> client.? 10.177.32.66:0/25020608
>>>>>>>>> 2011-12-01 20:18:17.143125 7fe03223b700 mon.0@0(leader).auth v146
>>>>>>>>> preprocess_query auth(proto 0 21 bytes) v1 from client.?
>>>>>>>>> 10.177.32.66:0/25020608
>>>>>>>>> 2011-12-01 20:18:17.143168 7fe03223b700 -- 10.177.64.4:6789/0 -->
>>>>>>>>> 10.177.32.66:0/25020608 -- auth_reply(proto 0 -95 Operation not
>>>>>>>>> supported) v1 -- ?+0 0x2f30600 con 0x297adc0
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Here you can see qemu isn't authenticating correctly, probably
>>>>>>>> because
>>>>>>>> the
>>>>>>>> rados user isn't being set by libvirt. With your libvirt version,
>>>>>>>> you'll
>>>>>>>> need to change:
>>>>>>>>
>>>>>>>>
>>>>>>>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
>>>>>>>>
>>>>>>>> to
>>>>>>>>
>>>>>>>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1:id=foo'>
>>>>>>>>
>>>>>>>> to authenticate as client.foo. Any ceph config option can be set
>>>>>>>> like
>>>>>>>> that,
>>>>>>>> separated by colons.
>>>>>>>>
>>>>>>>> If this still doesn't work, I'd suggest setting
>>>>>>>> debug_rbd=20:debug_monc=20:debug_auth=20:log_to_stderr=2 to get more
>>>>>>>> info
>>>>>>>> in
>>>>>>>> the libvirt instance log (usually
>>>>>>>> /var/log/libvirt/qemu/instance_name.log).
>>>>>>>> There may be apparmour restrictions preventing qemu from reading the
>>>>>>>> keyring
>>>>>>>> or ceph config files.
>>>>>>>>
>>>>>>>>
>>>>>>>>> ceph.conf, and keyring.bin exists on server, and machine with
>>>>>>>>> qemu-kvm
>>>>>>>>> in same directory:
>>>>>>>>> ; global
>>>>>>>>> [global]
>>>>>>>>> ; enable secure authentication
>>>>>>>>>
>>>>>>>>> auth supported = cephx
>>>>>>>>> keyring = /etc/ceph/keyring.bin
>>>>>>>>>
>>>>>>>>> debug rgw = 1
>>>>>>>>>
>>>>>>>>> rgw print continue = false
>>>>>>>>> rgw socket path = /var/run/radosgw.sock
>>>>>>>>>
>>>>>>>>> ; monitors
>>>>>>>>> ; You need at least one. You need at least three if you want to
>>>>>>>>> ; tolerate any node failures. Always create an odd number.
>>>>>>>>> [mon]
>>>>>>>>> mon data = /vol0/data/mon.$id
>>>>>>>>>
>>>>>>>>> ; some minimal logging (just message traffic) to aid
>>>>>>>>> debugging
>>>>>>>>>
>>>>>>>>> debug ms = 1 ; see message traffic
>>>>>>>>> debug mon = 0 ; monitor
>>>>>>>>> debug paxos = 0 ; monitor replication
>>>>>>>>> debug auth = 0 ;
>>>>>>>>>
>>>>>>>>> mon allowed clock drift = 2
>>>>>>>>>
>>>>>>>>> [mon.0]
>>>>>>>>> host = 10-177-64-4
>>>>>>>>> mon addr = 10.177.64.4:6789
>>>>>>>>>
>>>>>>>>> ; radosgw client list
>>>>>>>>> [client.radosgw.10-177-64-4]
>>>>>>>>>
>>>>>>>>> host = 10-177-64-4
>>>>>>>>> log file = /var/log/ceph/$name.log
>>>>>>>>> debug rgw = 1
>>>>>>>>> debug ms = 1
>>>>>>>>>
>>>>>>>>> ; osd
>>>>>>>>> ; You need at least one. Two if you want data to be replicated.
>>>>>>>>> ; Define as many as you like.
>>>>>>>>> [osd]
>>>>>>>>> ; This is where the btrfs volume will be mounted.
>>>>>>>>>
>>>>>>>>> osd data = /vol0/data/osd.$id
>>>>>>>>>
>>>>>>>>> ; Ideally, make this a separate disk or partition. A few
>>>>>>>>> GB
>>>>>>>>> ; is usually enough; more if you have fast disks. You can
>>>>>>>>> use
>>>>>>>>> ; a file under the osd data dir if need be
>>>>>>>>> ; (e.g. /data/osd$id/journal), but it will be slower than a
>>>>>>>>> ; separate disk or partition.
>>>>>>>>>
>>>>>>>>> osd journal = /vol0/data/osd.$id/journal
>>>>>>>>>
>>>>>>>>> ; If the OSD journal is a file, you need to specify the
>>>>>>>>> size.
>>>>>>>>> This is specified in MB.
>>>>>>>>>
>>>>>>>>> osd journal size = 512
>>>>>>>>>
>>>>>>>>> filestore journal writeahead = 1
>>>>>>>>> osd heartbeat grace = 5
>>>>>>>>>
>>>>>>>>> debug ms = 0 ; message traffic
>>>>>>>>> debug osd = 0
>>>>>>>>> debug filestore = 0 ; local object storage
>>>>>>>>> debug journal = 0 ; local journaling
>>>>>>>>> debug monc = 0
>>>>>>>>> debug rados = 1
>>>>>>>>>
>>>>>>>>> [osd.0]
>>>>>>>>> host = 10-177-64-4
>>>>>>>>> osd data = /vol0/data/osd.0
>>>>>>>>> keyring = /vol0/data/osd.0/keyring
>>>>>>>>>
>>>>>>>>> [osd.1]
>>>>>>>>> host = 10-177-64-4
>>>>>>>>> osd data = /vol0/data/osd.1
>>>>>>>>> keyring = /vol0/data/osd.1/keyring
>>>>>>>>>
>>>>>>>>> [osd.2]
>>>>>>>>> host = 10-177-64-4
>>>>>>>>> osd data = /vol0/data/osd.2
>>>>>>>>> keyring = /vol0/data/osd.2/keyring
>>>>>>>>>
>>>>>>>>> [osd.3]
>>>>>>>>> host = 10-177-64-4
>>>>>>>>> osd data = /vol0/data/osd.3
>>>>>>>>> keyring = /vol0/data/osd.3/keyring
>>>>>>>>>
>>>>>>>>> [osd.4]
>>>>>>>>> host = 10-177-64-4
>>>>>>>>> osd data = /vol0/data/osd.4
>>>>>>>>> keyring = /vol0/data/osd.4/keyring
>>>>>>>>>
>>>>>>>>> [osd.5]
>>>>>>>>> host = 10-177-64-4
>>>>>>>>> osd data = /vol0/data/osd.5
>>>>>>>>> keyring = /vol0/data/osd.5/keyring
>>>>>>>>>
>>>>>>>>> [osd.6]
>>>>>>>>> host = 10-177-64-4
>>>>>>>>> osd data = /vol0/data/osd.6
>>>>>>>>> keyring = /vol0/data/osd.6/keyring
>>>>>>>>>
>>>>>>>>> [osd.7]
>>>>>>>>> host = 10-177-64-4
>>>>>>>>> osd data = /vol0/data/osd.7
>>>>>>>>> keyring = /vol0/data/osd.7/keyring
>>>>>>>>>
>>>>>>>>> [osd.8]
>>>>>>>>> host = 10-177-64-4
>>>>>>>>> osd data = /vol0/data/osd.8
>>>>>>>>> keyring = /vol0/data/osd.8/keyring
>>>>>>>>>
>>>>>>>>> [osd.9]
>>>>>>>>> host = 10-177-64-4
>>>>>>>>> osd data = /vol0/data/osd.9
>>>>>>>>> keyring = /vol0/data/osd.9/keyring
>>>>>>>>>
>>>>>>>>> [osd.10]
>>>>>>>>> host = 10-177-64-4
>>>>>>>>> osd data = /vol0/data/osd.10
>>>>>>>>> keyring = /vol0/data/osd.10/keyring
>>>>>>>>>
>>>>>>>>> [osd.11]
>>>>>>>>> host = 10-177-64-4
>>>>>>>>> osd data = /vol0/data/osd.11
>>>>>>>>> keyring = /vol0/data/osd.11/keyring
>>>>>>>>>
>>>>>>>>> [osd.12]
>>>>>>>>> host = 10-177-64-4
>>>>>>>>> osd data = /vol0/data/osd.12
>>>>>>>>> keyring = /vol0/data/osd.12/keyring
>>>>>>>>>
>>>>>>>>> [osd.13]
>>>>>>>>> host = 10-177-64-4
>>>>>>>>> osd data = /vol0/data/osd.13
>>>>>>>>> keyring = /vol0/data/osd.13/keyring
>>>>>>>>>
>>>>>>>>> [osd.14]
>>>>>>>>> host = 10-177-64-4
>>>>>>>>> osd data = /vol0/data/osd.14
>>>>>>>>> keyring = /vol0/data/osd.14/keyring
>>>>>>>>>
>>>>>>>>> [osd.15]
>>>>>>>>> host = 10-177-64-4
>>>>>>>>> osd data = /vol0/data/osd.15
>>>>>>>>> keyring = /vol0/data/osd.15/keyring
>>>>>>>>>
>>>>>>>>> [osd.16]
>>>>>>>>> host = 10-177-64-4
>>>>>>>>> osd data = /vol0/data/osd.16
>>>>>>>>> keyring = /vol0/data/osd.16/keyring
>>>>>>>>>
>>>>>>>>> [osd.17]
>>>>>>>>> host = 10-177-64-4
>>>>>>>>> osd data = /vol0/data/osd.17
>>>>>>>>> keyring = /vol0/data/osd.17/keyring
>>>>>>>>>
>>>>>>>>> [osd.18]
>>>>>>>>> host = 10-177-64-4
>>>>>>>>> osd data = /vol0/data/osd.18
>>>>>>>>> keyring = /vol0/data/osd.18/keyring
>>>>>>>>>
>>>>>>>>> [osd.19]
>>>>>>>>> host = 10-177-64-4
>>>>>>>>> osd data = /vol0/data/osd.19
>>>>>>>>> keyring = /vol0/data/osd.19/keyring
>>>>>>>>>
>>>>>>>>> [osd.20]
>>>>>>>>> host = 10-177-64-4
>>>>>>>>> osd data = /vol0/data/osd.20
>>>>>>>>> keyring = /vol0/data/osd.20/keyring
>>>>>>>>>
>>>>>>>>> [osd.21]
>>>>>>>>> host = 10-177-64-4
>>>>>>>>> osd data = /vol0/data/osd.21
>>>>>>>>> keyring = /vol0/data/osd.21/keyring
>>>>>>>>>
>>>>>>>>> [osd.22]
>>>>>>>>> host = 10-177-64-4
>>>>>>>>> osd data = /vol0/data/osd.22
>>>>>>>>> keyring = /vol0/data/osd.22/keyring
>>>>>>>>>
>>>>>>>>> [osd.23]
>>>>>>>>> host = 10-177-64-4
>>>>>>>>> osd data = /vol0/data/osd.23
>>>>>>>>> keyring = /vol0/data/osd.23/keyring
>>>>>>>>>
>>>>>>>>> [osd.24]
>>>>>>>>> host = 10-177-64-4
>>>>>>>>> osd data = /vol0/data/osd.24
>>>>>>>>> keyring = /vol0/data/osd.24/keyring
>>>>>>>>>
>>>>>>>>> [osd.25]
>>>>>>>>> host = 10-177-64-4
>>>>>>>>> osd data = /vol0/data/osd.25
>>>>>>>>> keyring = /vol0/data/osd.25/keyring
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> -----
>>> Pozdrawiam
>>>
>>> Sławek "sZiBis" Skowron
>>
>>
>>
>>
>
--
-----
Pozdrawiam
Sławek "sZiBis" Skowron
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2011-12-20 9:38 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-12-01 19:37 Problem with attaching rbd device in qemu-kvm Sławomir Skowron
2011-12-01 20:42 ` Josh Durgin
2011-12-09 12:48 ` Sławomir Skowron
2011-12-12 18:53 ` Josh Durgin
2011-12-13 12:56 ` Sławomir Skowron
2011-12-13 20:08 ` Josh Durgin
2011-12-16 12:17 ` Sławomir Skowron
2011-12-19 15:37 ` Sławomir Skowron
2011-12-20 2:41 ` Josh Durgin
2011-12-20 9:38 ` Sławomir Skowron
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.