rbd - input/output error

rbd - input/output error

[Mario van der Linde]

[-- Attachment #1: Type: text/plain, Size: 2621 bytes --]

Hello,

i'm playing around with ceph and set up a cluster with 5 nodes. cephfs
is working fine, but if i try to use rbd i get the following error:

> ceph1:~ # modprobe rbd
> ceph1:~ # rbd create testimg --size 10240
> create error: error 5: Input/output error
> ceph1:~ # ceph --version
> ceph version 0.37 (commit:a6f3bbb744a6faea95ae48317f0b838edb16a896)

There are no log messages that give me a hint whats going wrong.

Does anybody know what the cause of this error is?

Here is the ceph.conf used for this cluster:
> [global]
>         log dir = /srv/ceph/out
>         logger dir = /srv/ceph/log
>        ; chdir = ""
>         pid file = /srv/ceph/out/$type$id.pid
> [mds]
>         pid file = /srv/ceph/out/$name.pid
>        ; lockdep = 1
>        ; debug ms = 1
>        ; debug mds = 20
>        ; mds log max segments = 2
>        max mds = 3
> [osd]
>        ; lockdep = 1
>        ; debug ms = 1
>        ; debug osd = 25
>        ; debug journal = 20
>        ; debug filestore = 10
>         osd data = /srv/ceph/dev/osd$id
>         osd journal = /srv/ceph/dev/osd$id/journal
>         osd journal size = 512
> [mon]
>        lockdep = 1
>        debug mon = 20
>        debug paxos = 20
>        debug ms = 1
>        mon data = "/srv/ceph/dev/mon$id"
> [mon.1]
>        host = ceph1
>        mon addr = 192.168.1.221:6789
> [mon.2]
>        host = ceph2
>        mon addr = 192.168.1.222:6789
> [mon.3]
>        host = ceph3
>        mon addr = 192.168.1.223:6789
> ;[mon.4]
> ;       host = ceph4
> ;       mon addr = 192.168.1.224:6789
> ;[mon.5]
> ;       host = ceph5
> ;       mon addr = 192.168.1.225:6789
> [osd.1]
>        host = ceph1
>        btrfs devs = /dev/sda6
> [osd.2]
>        host = ceph2
>        btrfs devs = /dev/sda6
> [osd.3]
>        host = ceph3
>        btrfs devs = /dev/sda6
> [osd.4]
>        host = ceph4
>        btrfs devs = /dev/sda6
> [osd.5]
>        host = ceph5
>        btrfs devs = /dev/sda6
> [mds.0]
>        host = ceph1
> [mds.1]
>        host = ceph2
> [mds.2]
>        host = ceph3


Greetings
Mario

-- 

Mario van der Linde                  Geschaeftsfuehrer: Ralph Dehner
Linux Consultant & Trainer           Unternehmenssitz:  Vohburg
B1 Systems GmbH                      Amtsgericht:       Ingolstadt
Osterfeldstraße 7                    Handelsregister:   HRB 3537
85088 Vohburg                        http://www.b1-systems.de
Mobil: +49 151 168 794 85
EMail: linde@b1-systems.de

http://pgpkeys.pca.dfn.de/pks/lookup?op=get&search=0xD31B76884480BB04


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: rbd - input/output error

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3027 bytes --]

On Sun, 13 Nov 2011, Mario van der Linde wrote:

> Hello,
> 
> i'm playing around with ceph and set up a cluster with 5 nodes. cephfs
> is working fine, but if i try to use rbd i get the following error:
> 
> > ceph1:~ # modprobe rbd
> > ceph1:~ # rbd create testimg --size 10240
> > create error: error 5: Input/output error

Can you try with 

 rbd create testmsg --size 10240 --log-to-stderr --debug-ms 1

so we can see a bit more about what is going on?

Thanks-
sage



> > ceph1:~ # ceph --version
> > ceph version 0.37 (commit:a6f3bbb744a6faea95ae48317f0b838edb16a896)
> 
> There are no log messages that give me a hint whats going wrong.
> 
> Does anybody know what the cause of this error is?
> 
> Here is the ceph.conf used for this cluster:
> > [global]
> >         log dir = /srv/ceph/out
> >         logger dir = /srv/ceph/log
> >        ; chdir = ""
> >         pid file = /srv/ceph/out/$type$id.pid
> > [mds]
> >         pid file = /srv/ceph/out/$name.pid
> >        ; lockdep = 1
> >        ; debug ms = 1
> >        ; debug mds = 20
> >        ; mds log max segments = 2
> >        max mds = 3
> > [osd]
> >        ; lockdep = 1
> >        ; debug ms = 1
> >        ; debug osd = 25
> >        ; debug journal = 20
> >        ; debug filestore = 10
> >         osd data = /srv/ceph/dev/osd$id
> >         osd journal = /srv/ceph/dev/osd$id/journal
> >         osd journal size = 512
> > [mon]
> >        lockdep = 1
> >        debug mon = 20
> >        debug paxos = 20
> >        debug ms = 1
> >        mon data = "/srv/ceph/dev/mon$id"
> > [mon.1]
> >        host = ceph1
> >        mon addr = 192.168.1.221:6789
> > [mon.2]
> >        host = ceph2
> >        mon addr = 192.168.1.222:6789
> > [mon.3]
> >        host = ceph3
> >        mon addr = 192.168.1.223:6789
> > ;[mon.4]
> > ;       host = ceph4
> > ;       mon addr = 192.168.1.224:6789
> > ;[mon.5]
> > ;       host = ceph5
> > ;       mon addr = 192.168.1.225:6789
> > [osd.1]
> >        host = ceph1
> >        btrfs devs = /dev/sda6
> > [osd.2]
> >        host = ceph2
> >        btrfs devs = /dev/sda6
> > [osd.3]
> >        host = ceph3
> >        btrfs devs = /dev/sda6
> > [osd.4]
> >        host = ceph4
> >        btrfs devs = /dev/sda6
> > [osd.5]
> >        host = ceph5
> >        btrfs devs = /dev/sda6
> > [mds.0]
> >        host = ceph1
> > [mds.1]
> >        host = ceph2
> > [mds.2]
> >        host = ceph3
> 
> 
> Greetings
> Mario
> 
> -- 
> 
> Mario van der Linde                  Geschaeftsfuehrer: Ralph Dehner
> Linux Consultant & Trainer           Unternehmenssitz:  Vohburg
> B1 Systems GmbH                      Amtsgericht:       Ingolstadt
> Osterfeldstraße 7                    Handelsregister:   HRB 3537
> 85088 Vohburg                        http://www.b1-systems.de
> Mobil: +49 151 168 794 85
> EMail: linde@b1-systems.de
> 
> http://pgpkeys.pca.dfn.de/pks/lookup?op=get&search=0xD31B76884480BB04
> 
> 

Re: rbd - input/output error

[Mario van der Linde]

[-- Attachment #1.1: Type: text/plain, Size: 1258 bytes --]

Hello,

On 14.11.2011 20:48, Sage Weil wrote:
> On Sun, 13 Nov 2011, Mario van der Linde wrote:
[...]
>>> ceph1:~ # modprobe rbd
>>> ceph1:~ # rbd create testimg --size 10240
>>> create error: error 5: Input/output error
> 
> Can you try with 
> 
>  rbd create testmsg --size 10240 --log-to-stderr --debug-ms 1
> 
> so we can see a bit more about what is going on?

This one gave me some output:
rbd create testimg --size=10240 --log-to-stderr 2 --debug-ms 2

See attached file.

Additional Info:
> ceph5:~ # modinfo rbd
> filename:       /lib/modules/3.1.0-rc9-1-desktop/kernel/drivers/block/rbd.ko
> license:        GPL
> author:         Jeff Garzik <jeff@garzik.org>
> description:    rados block device
> author:         Yehuda Sadeh <yehuda@hq.newdream.net>
> author:         Sage Weil <sage@newdream.net>
> srcversion:     C8F4E38D4A716AB32B9F75C
> depends:        
> vermagic:       3.1.0-rc9-1-desktop SMP preempt mod_unload modversions


Regards
Mario


-- 
Mario van der Linde
Linux Consultant & Trainer
Tel.: +49-151-16879485
Mail: linde@b1-systems.de

B1 Systems GmbH
Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de
GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537

[-- Attachment #1.2: ceph_failure.out --]
[-- Type: text/plain, Size: 5760 bytes --]

ceph5:~ # rbd create testimg --size=10240 --log-to-stderr 2 --debug-ms 2
2011-11-14 22:22:01.046036 7fa0dff51720 -- :/0 messenger.start
2011-11-14 22:22:01.046320 7fa0dff51720 -- :/1007622 --> 192.168.1.222:6789/0 -- auth(proto 0 30 bytes) v1 -- ?+0 0x7a60f0 con 0x7a5e80
2011-11-14 22:22:01.046576 7fa0dff4f700 -- :/1007622 >> 192.168.1.222:6789/0 pipe(0x7a5c10 sd=3 pgs=0 cs=0 l=0).connect error 192.168.1.222:6789/0, 111: Connection refused
2011-11-14 22:22:01.046616 7fa0dff4f700 -- :/1007622 >> 192.168.1.222:6789/0 pipe(0x7a5c10 sd=3 pgs=0 cs=0 l=0).fault 111: Connection refused
2011-11-14 22:22:01.046659 7fa0dff4f700 -- :/1007622 >> 192.168.1.222:6789/0 pipe(0x7a5c10 sd=3 pgs=0 cs=0 l=0).fault first fault
2011-11-14 22:22:01.046962 7fa0dff4f700 -- :/1007622 >> 192.168.1.222:6789/0 pipe(0x7a5c10 sd=3 pgs=0 cs=0 l=0).connect error 192.168.1.222:6789/0, 111: Connection refused
2011-11-14 22:22:01.046977 7fa0dff4f700 -- :/1007622 >> 192.168.1.222:6789/0 pipe(0x7a5c10 sd=3 pgs=0 cs=0 l=0).fault 111: Connection refused
2011-11-14 22:22:01.247422 7fa0dff4f700 -- :/1007622 >> 192.168.1.222:6789/0 pipe(0x7a5c10 sd=3 pgs=0 cs=0 l=0).connect error 192.168.1.222:6789/0, 111: Connection refused
2011-11-14 22:22:01.247448 7fa0dff4f700 -- :/1007622 >> 192.168.1.222:6789/0 pipe(0x7a5c10 sd=3 pgs=0 cs=0 l=0).fault 111: Connection refused
2011-11-14 22:22:01.647881 7fa0dff4f700 -- :/1007622 >> 192.168.1.222:6789/0 pipe(0x7a5c10 sd=3 pgs=0 cs=0 l=0).connect error 192.168.1.222:6789/0, 111: Connection refused
2011-11-14 22:22:01.647923 7fa0dff4f700 -- :/1007622 >> 192.168.1.222:6789/0 pipe(0x7a5c10 sd=3 pgs=0 cs=0 l=0).fault 111: Connection refused
2011-11-14 22:22:02.448373 7fa0dff4f700 -- :/1007622 >> 192.168.1.222:6789/0 pipe(0x7a5c10 sd=3 pgs=0 cs=0 l=0).connect error 192.168.1.222:6789/0, 111: Connection refused
2011-11-14 22:22:02.448414 7fa0dff4f700 -- :/1007622 >> 192.168.1.222:6789/0 pipe(0x7a5c10 sd=3 pgs=0 cs=0 l=0).fault 111: Connection refused
2011-11-14 22:22:04.046398 7fa0db8db700 -- :/1007622 mark_down 0x7a5e80 -- 0x7a5c10
2011-11-14 22:22:04.046564 7fa0db8db700 -- :/1007622 --> 192.168.1.221:6789/0 -- auth(proto 0 30 bytes) v1 -- ?+0 0x7a6a70 con 0x7a67b0
2011-11-14 22:22:04.047314 7fa0da8d9700 -- 192.168.1.225:0/1007622 learned my addr 192.168.1.225:0/1007622
2011-11-14 22:22:05.051649 7fa0dc8dd700 -- 192.168.1.225:0/1007622 <== mon.0 192.168.1.221:6789/0 1 ==== auth_reply(proto 1 0 Success) v1 ==== 24+0+0 (538567308 0 0) 0x7a6a70 con 0x7a67b0
2011-11-14 22:22:05.051723 7fa0dc8dd700 -- 192.168.1.225:0/1007622 --> 192.168.1.221:6789/0 -- mon_subscribe({monmap=0+}) v1 -- ?+0 0x7a60f0 con 0x7a67b0
2011-11-14 22:22:05.051953 7fa0dff51720 -- 192.168.1.225:0/1007622 --> 192.168.1.221:6789/0 -- mon_subscribe({monmap=0+,osdmap=0}) v1 -- ?+0 0x7a5c10 con 0x7a67b0
2011-11-14 22:22:05.051994 7fa0dff51720 -- 192.168.1.225:0/1007622 --> 192.168.1.221:6789/0 -- mon_subscribe({monmap=0+,osdmap=0}) v1 -- ?+0 0x7a6cf0 con 0x7a67b0
2011-11-14 22:22:05.052966 7fa0dc8dd700 -- 192.168.1.225:0/1007622 <== mon.0 192.168.1.221:6789/0 2 ==== mon_map v1 ==== 469+0+0 (2361102807 0 0) 0x7a5df0 con 0x7a67b0
2011-11-14 22:22:05.053015 7fa0dc8dd700 -- 192.168.1.225:0/1007622 <== mon.0 192.168.1.221:6789/0 3 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (2408960605 0 0) 0x7a60f0 con 0x7a67b0
2011-11-14 22:22:05.053206 7fa0dc8dd700 -- 192.168.1.225:0/1007622 <== mon.0 192.168.1.221:6789/0 4 ==== mon_map v1 ==== 469+0+0 (2361102807 0 0) 0x7a5df0 con 0x7a67b0
2011-11-14 22:22:05.053256 7fa0dc8dd700 -- 192.168.1.225:0/1007622 <== mon.0 192.168.1.221:6789/0 5 ==== osd_map(52..52 src has 1..52) v2 ==== 3463+0+0 (777870650 0 0) 0x7a7b70 con 0x7a67b0
2011-11-14 22:22:05.053462 7fa0dc8dd700 -- 192.168.1.225:0/1007622 <== mon.0 192.168.1.221:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (2408960605 0 0) 0x7a5c10 con 0x7a67b0
2011-11-14 22:22:05.053490 7fa0dc8dd700 -- 192.168.1.225:0/1007622 <== mon.0 192.168.1.221:6789/0 7 ==== mon_map v1 ==== 469+0+0 (2361102807 0 0) 0x7a7d90 con 0x7a67b0
2011-11-14 22:22:05.053767 7fa0dff51720 -- 192.168.1.225:0/1007622 --> 192.168.1.224:6800/3212 -- osd_op(client.4236.0:1 testimg.rbd [stat 0~0] 2.e35ad169) v1 -- ?+0 0x7a6d20 con 0x7a9ad0
2011-11-14 22:22:05.053908 7fa0dc8dd700 -- 192.168.1.225:0/1007622 <== mon.0 192.168.1.221:6789/0 8 ==== osd_map(52..52 src has 1..52) v2 ==== 3463+0+0 (777870650 0 0) 0x7a7000 con 0x7a67b0
2011-11-14 22:22:05.053959 7fa0dc8dd700 -- 192.168.1.225:0/1007622 <== mon.0 192.168.1.221:6789/0 9 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (2408960605 0 0) 0x7a9e80 con 0x7a67b0
2011-11-14 22:22:05.055250 7fa0dc8dd700 -- 192.168.1.225:0/1007622 <== osd.4 192.168.1.224:6800/3212 1 ==== osd_op_reply(1 testimg.rbd [stat 0~0] = -2 (No such file or directory)) v2 ==== 102+0+0 (1670433998 0 0) 0x7a6d20 con 0x7a9ad0
2011-11-14 22:22:05.055368 7fa0dff51720 -- 192.168.1.225:0/1007622 --> 192.168.1.224:6800/3212 -- osd_op(client.4236.0:2 rbd_info [write 0~0] 2.ac0e573a) v1 -- ?+0 0x7a6fa0 con 0x7a9ad0
2011-11-14 22:22:05.069088 7fa0dc8dd700 -- 192.168.1.225:0/1007622 <== osd.4 192.168.1.224:6800/3212 2 ==== osd_op_reply(2 rbd_info [write 0~0] ack = 0) v2 ==== 99+0+0 (3424702940 0 0) 0x7a6d20 con 0x7a9ad0
2011-11-14 22:22:05.069182 7fa0dff51720 -- 192.168.1.225:0/1007622 --> 192.168.1.224:6800/3212 -- osd_op(client.4236.0:3 rbd_info [call rbd.assign_bid] 2.ac0e573a) v1 -- ?+0 0x7a5cd0 con 0x7a9ad0
2011-11-14 22:22:05.069655 7fa0dc8dd700 -- 192.168.1.225:0/1007622 <== osd.4 192.168.1.224:6800/3212 3 ==== osd_op_reply(3 rbd_info [call] = -5 (Input/output error)) v2 ==== 99+0+0 (559231618 0 0) 0x7a6d20 con 0x7a9ad0
2011-11-14 22:22:05.069712 7fa0dff51720 librbd: failed to assign a block name for image
create error: error 5: Input/output error

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: rbd - input/output error

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1602 bytes --]

On Mon, 14 Nov 2011, Mario van der Linde wrote:
> Hello,
> 
> On 14.11.2011 20:48, Sage Weil wrote:
> > On Sun, 13 Nov 2011, Mario van der Linde wrote:
> [...]
> >>> ceph1:~ # modprobe rbd
> >>> ceph1:~ # rbd create testimg --size 10240
> >>> create error: error 5: Input/output error
> > 
> > Can you try with 
> > 
> >  rbd create testmsg --size 10240 --log-to-stderr --debug-ms 1
> > 
> > so we can see a bit more about what is going on?
> 
> This one gave me some output:
> rbd create testimg --size=10240 --log-to-stderr 2 --debug-ms 2
> 
> See attached file.

It sounds like the OSD can't find the libcls_rbd.so file.  Can you verify 
that it is in /usr/lib/rados-classes, and that that location isn't be 
overridden by something in ceph.conf?

sage


> 
> Additional Info:
> > ceph5:~ # modinfo rbd
> > filename:       /lib/modules/3.1.0-rc9-1-desktop/kernel/drivers/block/rbd.ko
> > license:        GPL
> > author:         Jeff Garzik <jeff@garzik.org>
> > description:    rados block device
> > author:         Yehuda Sadeh <yehuda@hq.newdream.net>
> > author:         Sage Weil <sage@newdream.net>
> > srcversion:     C8F4E38D4A716AB32B9F75C
> > depends:        
> > vermagic:       3.1.0-rc9-1-desktop SMP preempt mod_unload modversions
> 
> 
> Regards
> Mario
> 
> 
> -- 
> Mario van der Linde
> Linux Consultant & Trainer
> Tel.: +49-151-16879485
> Mail: linde@b1-systems.de
> 
> B1 Systems GmbH
> Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de
> GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
> 

Re: rbd - input/output error

[Mario van der Linde]

[-- Attachment #1: Type: text/plain, Size: 1544 bytes --]

Hello,

On 14.11.2011 22:39, Sage Weil wrote:
[...]
>> This one gave me some output:
>> rbd create testimg --size=10240 --log-to-stderr 2 --debug-ms 2
>>
>> See attached file.
> 
> It sounds like the OSD can't find the libcls_rbd.so file.  Can you verify 
> that it is in /usr/lib/rados-classes, and that that location isn't be 
> overridden by something in ceph.conf?

At first, on all Systems this file exists, but in /usr/lib64, not /usr/lib:
> ceph5:~ # for f in `seq 1 5`; do ssh ceph${f} ls -l /usr/lib64/rados-classes; done | grep ' libcls_rbd.so '
> lrwxrwxrwx 1 root root    19 Nov 10 10:15 libcls_rbd.so -> libcls_rbd.so.1.0.0
> lrwxrwxrwx 1 root root    19 Nov 10 10:15 libcls_rbd.so -> libcls_rbd.so.1.0.0
> lrwxrwxrwx 1 root root    19 Nov 10 10:16 libcls_rbd.so -> libcls_rbd.so.1.0.0
> lrwxrwxrwx 1 root root    19 Nov 10 10:16 libcls_rbd.so -> libcls_rbd.so.1.0.0
> lrwxrwxrwx 1 root root    19 Nov 10 10:16 libcls_rbd.so -> libcls_rbd.so.1.0.0

Seems to be the right place in an x86_64 system.

In ceph.conf there is no setting regarding to lib path.

Is ceph-radosgw needed for rbd? I think not, while radosgw is AFAIK the
REST interface...
This package is not installed (due to a not resolveable cgi dependency).


Regards
Mario


-- 
Mario van der Linde
Linux Consultant & Trainer
Tel.: +49-151-16879485
Mail: linde@b1-systems.de

B1 Systems GmbH
Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de
GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: rbd - input/output error

[Tommi Virtanen]

On Mon, Nov 14, 2011 at 13:59, Mario van der Linde <linde@b1-systems.de> wrote:
> At first, on all Systems this file exists, but in /usr/lib64, not /usr/lib:
>> ceph5:~ # for f in `seq 1 5`; do ssh ceph${f} ls -l /usr/lib64/rados-classes; done | grep ' libcls_rbd.so '
>> lrwxrwxrwx 1 root root    19 Nov 10 10:15 libcls_rbd.so -> libcls_rbd.so.1.0.0
>> lrwxrwxrwx 1 root root    19 Nov 10 10:15 libcls_rbd.so -> libcls_rbd.so.1.0.0
>> lrwxrwxrwx 1 root root    19 Nov 10 10:16 libcls_rbd.so -> libcls_rbd.so.1.0.0
>> lrwxrwxrwx 1 root root    19 Nov 10 10:16 libcls_rbd.so -> libcls_rbd.so.1.0.0
>> lrwxrwxrwx 1 root root    19 Nov 10 10:16 libcls_rbd.so -> libcls_rbd.so.1.0.0
>
> Seems to be the right place in an x86_64 system.
>
> In ceph.conf there is no setting regarding to lib path.

Ah, new-style multi-arch. We haven't tested with an environment like that yet.

You should be able to put this in ceph.conf and work around it:

[global]
osd class dir = /usr/lib64/rados-classes

> Is ceph-radosgw needed for rbd? I think not, while radosgw is AFAIK the
> REST interface...
> This package is not installed (due to a not resolveable cgi dependency).

No, it is not.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rbd - input/output error

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2195 bytes --]

On Mon, 14 Nov 2011, Mario van der Linde wrote:
> Hello,
> 
> On 14.11.2011 22:39, Sage Weil wrote:
> [...]
> >> This one gave me some output:
> >> rbd create testimg --size=10240 --log-to-stderr 2 --debug-ms 2
> >>
> >> See attached file.
> > 
> > It sounds like the OSD can't find the libcls_rbd.so file.  Can you verify 
> > that it is in /usr/lib/rados-classes, and that that location isn't be 
> > overridden by something in ceph.conf?
> 
> At first, on all Systems this file exists, but in /usr/lib64, not /usr/lib:
> > ceph5:~ # for f in `seq 1 5`; do ssh ceph${f} ls -l /usr/lib64/rados-classes; done | grep ' libcls_rbd.so '
> > lrwxrwxrwx 1 root root    19 Nov 10 10:15 libcls_rbd.so -> libcls_rbd.so.1.0.0
> > lrwxrwxrwx 1 root root    19 Nov 10 10:15 libcls_rbd.so -> libcls_rbd.so.1.0.0
> > lrwxrwxrwx 1 root root    19 Nov 10 10:16 libcls_rbd.so -> libcls_rbd.so.1.0.0
> > lrwxrwxrwx 1 root root    19 Nov 10 10:16 libcls_rbd.so -> libcls_rbd.so.1.0.0
> > lrwxrwxrwx 1 root root    19 Nov 10 10:16 libcls_rbd.so -> libcls_rbd.so.1.0.0
> 
> Seems to be the right place in an x86_64 system.

Argh... For now, just add
	osd class dir = /usr/lib64/rados-classes
and that will sort it out.  

My last attempt to make automake fill in the @libdirpath@ correctly in 
common/config_opts.h failed miserably, further entrenching my frustration 
with autotools.  It's bug http://tracker.newdream.net/issues/1614, if 
someone with better autotools-fu than I wants to fix this.  IIRC I tried a 
few different methods but could only make it substitute in @prefix@/lib64, 
but not /usr/lib64...

sage

> 
> In ceph.conf there is no setting regarding to lib path.
> 
> Is ceph-radosgw needed for rbd? I think not, while radosgw is AFAIK the
> REST interface...
> This package is not installed (due to a not resolveable cgi dependency).
> 
> 
> Regards
> Mario
> 
> 
> -- 
> Mario van der Linde
> Linux Consultant & Trainer
> Tel.: +49-151-16879485
> Mail: linde@b1-systems.de
> 
> B1 Systems GmbH
> Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de
> GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
> 
> 

Re: rbd - input/output error

[Tommi Virtanen]

On Mon, Nov 14, 2011 at 14:09, Sage Weil <sage@newdream.net> wrote:
...
> My last attempt to make automake fill in the @libdirpath@ correctly in
> common/config_opts.h failed miserably, further entrenching my frustration
> with autotools.  It's bug http://tracker.newdream.net/issues/1614, if
> someone with better autotools-fu than I wants to fix this.  IIRC I tried a
> few different methods but could only make it substitute in @prefix@/lib64,
> but not /usr/lib64...

For some reason #1614 didn't want me to edit it, so I opened
http://tracker.newdream.net/issues/1722 to make sure this doesn't get
lost. With all the new distributions coming out multi-arch, this is
going to get more common.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [solved] rbd - input/output error

[Mario van der Linde]

[-- Attachment #1: Type: text/plain, Size: 758 bytes --]

Hello,

On 14.11.2011 23:05, Tommi Virtanen wrote:
> On Mon, Nov 14, 2011 at 13:59, Mario van der Linde <linde@b1-systems.de> wrote:
>> At first, on all Systems this file exists, but in /usr/lib64, not /usr/lib:
[...]
> 
> Ah, new-style multi-arch. We haven't tested with an environment like that yet.
> 
> You should be able to put this in ceph.conf and work around it:
> 
> [global]
> osd class dir = /usr/lib64/rados-classes

This fixed it magically :)


Thank you
Mario


-- 
Mario van der Linde
Linux Consultant & Trainer
Tel.: +49-151-16879485
Mail: linde@b1-systems.de

B1 Systems GmbH
Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de
GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]