Re: installation: where do I start debugging this error?

[Jeff Wu <cpwu@tnsoft.com.cn>]

在 2010-12-08三的 09:53 +0800，Brian Chrisman写道：
> I've built and installed RPMs for ceph for RHEL6beta.
> I've placed the below ceph.conf in /etc/ceph on both of my test nodes
> (test10, test11).
> 
> I build a ceph filesystem and hide the key
> mkcephfs -c /etc/ceph/ceph.conf -a --mkbtrfs -k /etc/ceph/keyring.bin
> cauthtool --print-key /etc/ceph/keyring.bin > /etc/ceph/secret
> chmod 600 /etc/ceph/secret
> scp -p /etc/ceph/secret test11:/etc/ceph
> 
> Then I start the daemons on each node:
> service ceph start
> 
> My daemons start up on both nodes with 'service ceph -a start'
> root      3199     1  0 17:18 ?        00:00:00 /usr/local/bin/cmon -i
> 0 -c /tmp/ceph.conf.5365
> root      3228     1  0 17:18 ?        00:00:00 /usr/local/bin/cmds -i
> test10 -c /tmp/ceph.conf.5365
> root      3285     1  0 17:18 ?        00:00:00 /usr/local/bin/cosd -i
> 0 -c /tmp/ceph.conf.5365
> (similar output on other node)
> 
> I attempt to mount the ceph filesystem on test10 (using test11's IP):
> mount -t ceph -o name=admin,secretfile=/etc/ceph/keyring.bin
> 10.200.98.111:/ /mnt/ceph
> mount error 5 = Input/output error
> 

Hi ,i take the followng steps , fail to mount,too.

## save auth secret to a file:
$cauthtool --print-key /etc/ceph/keyring.bin > /etc/ceph/secret
chmod 600 /etc/ceph/secret
## future ,support,read a secret from a file
$ mount -t ceph 172.16.50.10:6789:/foo /mnt/ceph  
   -o name=admin,secretfile=secret

but ,the following steps ,mount ceph is successful.


# enable cephx, add a user and secret
    $ mount -t ceph -o name=admin,secret=<secret> 
       1.2.3.4:/ /mnt/ceph
$ mount -t ceph 172.16.50.10:6791:/foo /mnt/ceph 
    -o name=admin,secret=‘AQArWtdMiI1uDRAAVbNRMeiwsjK+DEMeB7ewLg==’

# cauthtool --list keyring.bin 
client.admin
 key: AQArWtdMiI1uDRAAVbNRMeiwsjK+DEMeB7ewLg==
auid: 0
caps: [mds] allow
caps: [mon] allow *
caps: [osd] allow *





> /var/log/messages seems to show me what the problem is:
> Dec  7 17:45:19 test10 kernel: libceph: mon0 10.200.98.111:6789
> connection failed
> (a few more of those before mount fails)
> on test11, the daemon is up and listening on that port:
> tcp        0      0 10.200.98.111:6789          10.200.98.111:56805
>      ESTABLISHED 7781/cmon
> 
> And here's /var/log/ceph/mon.1.log on test11 (the .111 node)
> 2010-12-07 17:36:20.136921 --- 7780 opened log /var/log/ceph/mon.1.log ---
> ceph version 0.24~rc (commit:378d13df9505e4ea9a32f42cb713cdcf7aaccda0)
> 2010-12-07 17:36:20.137164 7f80c51e3720 store(/data/mon1) mount
> 2010-12-07 17:36:20.138241 7f80c51e3720 mon.1@1(starting) e1 init fsid
> 1b4cabdb-30d2-752d-005f-517a7fa982f8
> 2010-12-07 17:36:20.165407 7f80c51e3720 log [INF] : mon.1 calling new
> monitor election
> 2010-12-07 17:36:20.192343 7f80c51e1710 -- 10.200.98.111:6789/0 >>
> 10.200.98.110:6789/0 pipe(0x1cafd20 sd=6 pgs=0 cs=0 l=0).fault first
> fault
> 
> And for test10 (the .110 node)
> 2010-12-07 17:36:49.183357 --- 5767 opened log /var/log/ceph/mon.0.log ---
> ceph version 0.24~rc (commit:378d13df9505e4ea9a32f42cb713cdcf7aaccda0)
> 2010-12-07 17:36:49.183545 7ff24669c720 store(/data/mon0) mount
> 2010-12-07 17:36:49.184556 7ff24669c720 mon.0@0(starting) e1 init fsid
> 1b4cabdb-30d2-752d-005f-517a7fa982f8
> 2010-12-07 17:36:49.600650 7ff24669c720 log [INF] : mon.0 calling new
> monitor election
> 2010-12-07 17:36:49.645875 7ff24669a710 -- 10.200.98.110:6789/0 >>
> 10.200.98.111:6789/0 pipe(0xac7d20 sd=6 pgs=0 cs=0 l=0).fault first
> fault
> 
> 
> Does this mean my cmon on 111 is getting into a state where it's not
> receiving incoming connections?
> Any suggestions on where to go from here?
> 
> thanks,
> Brian Chrisman
> 
> 
> 
> ----- ceph.conf in /etc/ceph -----
> ; From sample:
> [global]
> 	auth supported = cephx
> 
> [mon]
> 	mon data = /data/mon$id
> 
> 	; logging, for debugging monitor crashes, in order of
> 	; their likelihood of being helpful :)
> 	;debug ms = 1
> 	;debug mon = 20
> 	;debug paxos = 20
> 	;debug auth = 20
> 
> [mon0]
> 	host = test10
> 	mon addr = 10.200.98.110:6789
> 
> [mon1]
> 	host = test11
> 	mon addr = 10.200.98.111:6789
> 
> [mds]
> 	keyring = /data/keyring.$name
> 	;debug ms = 1
> 	;debug mds = 20
> 
> [mds.test10]
> 	host = test10
> 
> [mds.test11]
> 	host = test11
> 
> [osd]
> 	osd data = /data/osd$id
> 	osd journal = /data/osd$id/journal
> 	osd journal size = 1000 ; journal size, in megabytes
> 	;debug ms = 1
> 	;debug osd = 20
> 	;debug filestore = 20
> 	;debug journal = 20
> 
> [osd0]
> 	host = test10
> 	btrfs devs = /dev/sdd4
> 
> [osd1]
> 	host = test11
> 	btrfs devs = /dev/sdd4
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html