* Simple test system not working
@ 2011-02-01 6:48 Chris Dunlop
2011-02-01 6:56 ` Chris Dunlop
2011-02-01 23:01 ` Samuel Just
0 siblings, 2 replies; 6+ messages in thread
From: Chris Dunlop @ 2011-02-01 6:48 UTC (permalink / raw)
To: ceph-devel
G'day,
I'm just starting with ceph and am having trouble getting a test ceph
installation going on debian lenny with a self-compiled 2.6.37. Ceph
itself is latest unstable (73e76723) compiled using dpkg-buildpackage.
The mon, mds and osd are all on the same machine. The ceph base /data
directory is ext2 mounted with user_xattr. The ceph filesystem is
created using the attached ceph.conf and:
/sbin/mkcephfs -v -c /etc/ceph/ceph.conf --allhosts --mkbtrfs -k
/etc/ceph/keyring.bin
Once ceph has been started, trying a mount on a (separate) client
machine (2.6.37 kernel) hangs before eventually giving an error:
# mount -t ceph b3:/ /mnt
mount error 5 = Input/output error
On the server side, it seems that the cmon isn't accepting connections
from the mds or osd. E.g. repeated instances of this in the osd log,
and similar "mark_down"s in the mds log:
2011-02-01 17:18:47.463897 4a254950 -- 0.0.0.0:6801/7854 mark_down
10.200.63.129:6789/0 -- 0xca8a00
2011-02-01 17:18:47.463924 4a254950 -- 0.0.0.0:6801/7854 --> mon0
10.200.63.129:6789/0 -- auth(proto 0 26 bytes) v1 -- ?+0 0xcd9800
The mon log (amongst a whole bunch of other stuff) repeats:
2011-02-01 17:18:51.445785 42c73950 mon.0@0(leader).mds e1 e1: 0/0/1 up
2011-02-01 17:18:51.445790 42c73950 mon.0@0(leader).osd e1 e1: 0 osds: 0 up, 0 in
The attached logs.zip has the full logs from one attempted start.
Is there something obvious I'm doing wrong?
Thanks for any assistance!
Chris
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Simple test system not working
2011-02-01 6:48 Simple test system not working Chris Dunlop
@ 2011-02-01 6:56 ` Chris Dunlop
2011-02-01 23:01 ` Samuel Just
1 sibling, 0 replies; 6+ messages in thread
From: Chris Dunlop @ 2011-02-01 6:56 UTC (permalink / raw)
To: ceph-devel
Chris Dunlop <chris <at> onthe.net.au> writes:
> The ceph filesystem is created using the attached ceph.conf and:
Sorry, no attachments.
ceph.conf:
--------------------------------------------------
[global]
pid file = /var/run/ceph/$name.pid
debug ms = 20
debug mon = 20
[mon]
mon data = /data/mon$id
debug ms = 90
debug mon = 90
debug paxos = 90
debug auth = 90
[mon0]
host = b3
mon addr = 10.200.63.129:6789
[mds]
keyring = /data/keyring.$name
[mds.b3]
host = b3
[osd]
osd data = /data/osd$id
osd journal = /data/osd$id/journal
osd journal size = 1000 ; journal size, in megabytes
debug ms = 1
debug osd = 20
debug filestore = 20
debug journal = 20
[osd0]
host = b3
btrfs devs = /dev/sdb1
--------------------------------------------------
> The attached logs.zip has the full logs from one attempted start.
Available at: http://www.onthe.net.au/private/logs.zip
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Simple test system not working
2011-02-01 6:48 Simple test system not working Chris Dunlop
2011-02-01 6:56 ` Chris Dunlop
@ 2011-02-01 23:01 ` Samuel Just
2011-02-02 2:39 ` Chris Dunlop
1 sibling, 1 reply; 6+ messages in thread
From: Samuel Just @ 2011-02-01 23:01 UTC (permalink / raw)
Cc: ceph-devel
From the logs, it looks like the osd and mds are failing to bind to
an address. Perhaps b3 is not a valid host name?
-Sam
On 01/31/2011 10:48 PM, Chris Dunlop wrote:
> G'day,
>
> I'm just starting with ceph and am having trouble getting a test ceph
> installation going on debian lenny with a self-compiled 2.6.37. Ceph
> itself is latest unstable (73e76723) compiled using dpkg-buildpackage.
> The mon, mds and osd are all on the same machine. The ceph base /data
> directory is ext2 mounted with user_xattr. The ceph filesystem is
> created using the attached ceph.conf and:
>
> /sbin/mkcephfs -v -c /etc/ceph/ceph.conf --allhosts --mkbtrfs -k
> /etc/ceph/keyring.bin
>
> Once ceph has been started, trying a mount on a (separate) client
> machine (2.6.37 kernel) hangs before eventually giving an error:
>
> # mount -t ceph b3:/ /mnt
> mount error 5 = Input/output error
>
> On the server side, it seems that the cmon isn't accepting connections
> from the mds or osd. E.g. repeated instances of this in the osd log,
> and similar "mark_down"s in the mds log:
>
> 2011-02-01 17:18:47.463897 4a254950 -- 0.0.0.0:6801/7854 mark_down
> 10.200.63.129:6789/0 -- 0xca8a00
> 2011-02-01 17:18:47.463924 4a254950 -- 0.0.0.0:6801/7854 --> mon0
> 10.200.63.129:6789/0 -- auth(proto 0 26 bytes) v1 -- ?+0 0xcd9800
>
> The mon log (amongst a whole bunch of other stuff) repeats:
>
> 2011-02-01 17:18:51.445785 42c73950 mon.0@0(leader).mds e1 e1: 0/0/1 up
> 2011-02-01 17:18:51.445790 42c73950 mon.0@0(leader).osd e1 e1: 0 osds: 0 up, 0 in
>
> The attached logs.zip has the full logs from one attempted start.
>
> Is there something obvious I'm doing wrong?
>
> Thanks for any assistance!
>
> Chris
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Simple test system not working
2011-02-01 23:01 ` Samuel Just
@ 2011-02-02 2:39 ` Chris Dunlop
[not found] ` <AANLkTinHREewy2QUt8cC33YzaTdS56VENBV6m2-kUuoM@mail.gmail.com>
0 siblings, 1 reply; 6+ messages in thread
From: Chris Dunlop @ 2011-02-02 2:39 UTC (permalink / raw)
To: ceph-devel
Samuel Just <samuelj <at> hq.newdream.net> writes:
> From the logs, it looks like the osd and mds are failing to bind to
> an address. Perhaps b3 is not a valid host name?
Thanks!
That wasn't it (b3 is valid), but you started me looking at the
networking side of things again (for the nth time) and this time
around I noticed there was no loopback interface on the box. That in
itself was odd, although it didn't immediately spring to mind that the
lack of a loopback would stop ceph working. But I added a loopback
anyway on the basis of "avoid oddness" and that got it all going. I
still don't know why a loopback is required for ceph, but apparently
it is!
FYI, with it working, the osd and mds are still bound to the wildcard
address rather than to a specific address (as they were before the
loopback was installed). Without loopback:
# lsof -n -i | egrep 'COMMAND|cmon|cmds|cosd'
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
cmon 3321 root 5u IPv4 8451 TCP 10.200.63.129:6789 (LISTEN)
cmds 3357 root 4u IPv4 8465 TCP *:6800 (LISTEN)
cmds 3357 root 5u IPv4 10044 TCP
10.200.63.129:59480->10.200.63.129:6789 (SYN_SENT)
cosd 3466 root 3u IPv4 8548 TCP *:6801 (LISTEN)
cosd 3466 root 5u IPv4 8549 TCP *:6802 (LISTEN)
cosd 3466 root 6u IPv4 8550 TCP *:6803 (LISTEN)
cosd 3466 root 13u IPv4 10046 TCP
10.200.63.129:59481->10.200.63.129:6789 (SYN_SENT)
Note that the DEVICE number for the SYN_SENT connections keeps
changing in this situation.
With the loopback:
# lsof -n -i | egrep 'COMMAND|cmon|cmds|cosd'
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
cmon 6633 root 5u IPv4 13277 TCP 10.200.63.129:6789 (LISTEN)
cmon 6633 root 6u IPv4 14353 TCP
10.200.63.129:6789->10.200.63.129:32884 (ESTABLISHED)
cmon 6633 root 7u IPv4 13367 TCP
10.200.63.129:6789->10.200.63.129:32883 (ESTABLISHED)
cmds 6669 root 4u IPv4 13294 TCP *:6800 (LISTEN)
cmds 6669 root 5u IPv4 13366 TCP
10.200.63.129:32883->10.200.63.129:6789 (ESTABLISHED)
cmds 6669 root 10u IPv4 13456 TCP
10.200.63.129:41828->10.200.63.129:6801 (ESTABLISHED)
cosd 6741 root 3u IPv4 14350 TCP *:6801 (LISTEN)
cosd 6741 root 5u IPv4 14351 TCP *:6802 (LISTEN)
cosd 6741 root 6u IPv4 14352 TCP *:6803 (LISTEN)
cosd 6741 root 13u IPv4 13401 TCP
10.200.63.129:32884->10.200.63.129:6789 (ESTABLISHED)
cosd 6741 root 14u IPv4 14467 TCP
10.200.63.129:6801->10.200.63.129:41828 (ESTABLISHED)
Cheers,
Chris.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2011-02-02 5:36 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-02-01 6:48 Simple test system not working Chris Dunlop
2011-02-01 6:56 ` Chris Dunlop
2011-02-01 23:01 ` Samuel Just
2011-02-02 2:39 ` Chris Dunlop
[not found] ` <AANLkTinHREewy2QUt8cC33YzaTdS56VENBV6m2-kUuoM@mail.gmail.com>
2011-02-02 2:53 ` Fwd: " Neville Burnell
2011-02-02 5:35 ` Chris Dunlop
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.