mkcephfs giving problems

mkcephfs giving problems

[Atish Kathpal]

Hello 

What am I trying? -  To setup single node cluster for Ceph, OSD, MON and MDS on 
same node.
OS: Ubuntu 11.04 (natty)
I installed Ceph following these instructions: 
=================
wget -q -O- https://raw.github.com/NewDreamNetwork/ceph/master/keys/release.asc 
\
| sudo apt-key add -

sudo tee /etc/apt/sources.list.d/ceph.list <<EOF
deb http://ceph.newdream.net/debian/ DISTRO main
deb-src http://ceph.newdream.net/debian/ DISTRO main
EOF

sudo apt-get update
sudo apt-get install ceph
=================

Next I created /etc/ceph.conf file with following contents:
=========
[global]
        auth supported = cephx
       keyring = /etc/ceph/ceph.keyring

[mon]
        mon data = /ceph_data/mon.0

[mds]

[osd]
        osd data = /ceph_data/osd.0
        osd journal = /ceph_data/osd.0.journal
        osd journal size = 1000

[mon.0]
        host = atish-virtual-machine
        mon addr = 10.72.147.207:6789

[osd.0]
        host = atish-virtual-machine

[mds.0]
        host = atish-virtual-machine
===========

Now on running:  mkcephfs -a -c ceph.conf -k ceph.keyring

I get following output:
============
atish@atish-virtual-machine:/etc/ceph$ mkcephfs -a -c ceph.conf -k ceph.keyring
temp dir is /tmp/mkcephfs.dh5U7fAf0a
preparing monmap in /tmp/mkcephfs.dh5U7fAf0a/monmap
/usr/bin/monmaptool --create --clobber --add 0 10.72.147.207:6789 --print 
/tmp/mkcephfs.dh5U7fAf0a/monmap
/usr/bin/monmaptool: monmap file /tmp/mkcephfs.dh5U7fAf0a/monmap
/usr/bin/monmaptool: generated fsid 0fad0467-bfc6-5063-935b-5abd87dbebd5
epoch 1
fsid 0fad0467-bfc6-5063-935b-5abd87dbebd5
last_changed 2011-09-14 20:09:10.557121
created 2011-09-14 20:09:10.557121
0: 10.72.147.207:6789/0 mon.0
/usr/bin/monmaptool: writing epoch 1 to /tmp/mkcephfs.dh5U7fAf0a/monmap (1 
monitors)
=== osd.0 ===
failed to open log file '/var/log/ceph/osd.0.log': error 13: Permission denied
 ** WARNING: Ceph is still under development.  Any feedback can be directed  **
 **          at ceph-devel@vger.kernel.org or http://ceph.newdream.net/.     **
2011-09-14 20:09:10.944686 b77d86e0 AdminSocketConfigObs::init: failed: 
AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to 
'/var/run/ceph/osd.0.asok': error 2: No such file or directory
2011-09-14 20:09:10.944701 b77d86e0 AdminSocketConfigObs: failed to start 
AdminSocket
2011-09-14 20:09:10.946025 b77d86e0 filestore(/ceph_data/osd.0) FileStore::mkfs: 
failed to open /ceph_data/osd.0/fsid: error 13: Permission denied
2011-09-14 20:09:10.946036 b77d86e0 OSD::mkfs: FileStore::mkfs failed with error 
-13
2011-09-14 20:09:10.946243 b77d86e0  ** ERROR: error creating empty object store 
in /ceph_data/osd.0: error 13: Permission denied
failed: '/sbin/mkcephfs -d /tmp/mkcephfs.dh5U7fAf0a --init-daemon osd.0'
atish@atish-virtual-machine:/etc/ceph$
================

Can you please guide me with setting Ceph up correctly? Where have I gone wrong?

Final aim: I intend to use just the Ceph Object Storage - RASOS - with S3/REST.

Waiting for reply.

Regards
Atish

RE: mkcephfs giving problems

[Smets, Jan (Jan)]

 Try as root.

=>   failed to open log file '/var/log/ceph/osd.0.log': error 13: Permission denied



> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org 
> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Atish Kathpal
> Sent: donderdag 15 september 2011 8:14
> To: ceph-devel@vger.kernel.org
> Subject: mkcephfs giving problems
> 
> Hello 
> 
> What am I trying? -  To setup single node cluster for Ceph, 
> OSD, MON and MDS on 
> same node.
> OS: Ubuntu 11.04 (natty)
> I installed Ceph following these instructions: 
> =================
> wget -q -O- 
> https://raw.github.com/NewDreamNetwork/ceph/master/keys/release.asc 
> \
> | sudo apt-key add -
> 
> sudo tee /etc/apt/sources.list.d/ceph.list <<EOF
> deb http://ceph.newdream.net/debian/ DISTRO main
> deb-src http://ceph.newdream.net/debian/ DISTRO main
> EOF
> 
> sudo apt-get update
> sudo apt-get install ceph
> =================
> 
> Next I created /etc/ceph.conf file with following contents:
> =========
> [global]
>         auth supported = cephx
>        keyring = /etc/ceph/ceph.keyring
> 
> [mon]
>         mon data = /ceph_data/mon.0
> 
> [mds]
> 
> [osd]
>         osd data = /ceph_data/osd.0
>         osd journal = /ceph_data/osd.0.journal
>         osd journal size = 1000
> 
> [mon.0]
>         host = atish-virtual-machine
>         mon addr = 10.72.147.207:6789
> 
> [osd.0]
>         host = atish-virtual-machine
> 
> [mds.0]
>         host = atish-virtual-machine
> ===========
> 
> Now on running:  mkcephfs -a -c ceph.conf -k ceph.keyring
> 
> I get following output:
> ============
> atish@atish-virtual-machine:/etc/ceph$ mkcephfs -a -c 
> ceph.conf -k ceph.keyring
> temp dir is /tmp/mkcephfs.dh5U7fAf0a
> preparing monmap in /tmp/mkcephfs.dh5U7fAf0a/monmap
> /usr/bin/monmaptool --create --clobber --add 0 
> 10.72.147.207:6789 --print 
> /tmp/mkcephfs.dh5U7fAf0a/monmap
> /usr/bin/monmaptool: monmap file /tmp/mkcephfs.dh5U7fAf0a/monmap
> /usr/bin/monmaptool: generated fsid 
> 0fad0467-bfc6-5063-935b-5abd87dbebd5
> epoch 1
> fsid 0fad0467-bfc6-5063-935b-5abd87dbebd5
> last_changed 2011-09-14 20:09:10.557121
> created 2011-09-14 20:09:10.557121
> 0: 10.72.147.207:6789/0 mon.0
> /usr/bin/monmaptool: writing epoch 1 to 
> /tmp/mkcephfs.dh5U7fAf0a/monmap (1 
> monitors)
> === osd.0 ===
> failed to open log file '/var/log/ceph/osd.0.log': error 13: 
> Permission denied
>  ** WARNING: Ceph is still under development.  Any feedback 
> can be directed  **
>  **          at ceph-devel@vger.kernel.org or 
> http://ceph.newdream.net/.     **
> 2011-09-14 20:09:10.944686 b77d86e0 
> AdminSocketConfigObs::init: failed: 
> AdminSocket::bind_and_listen: failed to bind the UNIX domain 
> socket to 
> '/var/run/ceph/osd.0.asok': error 2: No such file or directory
> 2011-09-14 20:09:10.944701 b77d86e0 AdminSocketConfigObs: 
> failed to start 
> AdminSocket
> 2011-09-14 20:09:10.946025 b77d86e0 
> filestore(/ceph_data/osd.0) FileStore::mkfs: 
> failed to open /ceph_data/osd.0/fsid: error 13: Permission denied
> 2011-09-14 20:09:10.946036 b77d86e0 OSD::mkfs: 
> FileStore::mkfs failed with error 
> -13
> 2011-09-14 20:09:10.946243 b77d86e0  ** ERROR: error creating 
> empty object store 
> in /ceph_data/osd.0: error 13: Permission denied
> failed: '/sbin/mkcephfs -d /tmp/mkcephfs.dh5U7fAf0a 
> --init-daemon osd.0'
> atish@atish-virtual-machine:/etc/ceph$
> ================
> 
> Can you please guide me with setting Ceph up correctly? Where 
> have I gone wrong?
> 
> Final aim: I intend to use just the Ceph Object Storage - 
> RASOS - with S3/REST.
> 
> Waiting for reply.
> 
> Regards
> Atish
> 
> --
> To unsubscribe from this list: send the line "unsubscribe 
> ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: mkcephfs giving problems

[Wido den Hollander]

Hi,

As Jantje said, it seems you are not running as root.

But I do have some recommendations for your config:

On Thu, 2011-09-15 at 06:14 +0000, Atish Kathpal wrote:
> [global]
>         auth supported = cephx
>        keyring = /etc/ceph/ceph.keyring
> 
> [mon]
>         mon data = /ceph_data/mon.0
> 

This section is globally, so if you'd add more monitors, they would all
get /ceph_data/mon.0 as data dir.

I'd do:

[mon]
    mon data = /ceph_data/mon.$id

> [mds]
> 
> [osd]
>         osd data = /ceph_data/osd.0
>         osd journal = /ceph_data/osd.0.journal
>         osd journal size = 1000

Same goes here for your OSD, do:

  osd data = /ceph_data/osd.$id
  osd journal = /ceph_data/osd.$id.journal

> 
> [mon.0]
>         host = atish-virtual-machine
>         mon addr = 10.72.147.207:6789

Monitors should not be named "0", "1", etc, but more like:
* mon.alpha
* mon.beta
* mon.charlie

> 
> [osd.0]
>         host = atish-virtual-machine
> 
> [mds.0]
>         host = atish-virtual-machine


And you have only one OSD, is that correct?

Wido

Re: mkcephfs giving problems

[Wido den Hollander]

Hi,

On Thu, 2011-09-15 at 17:24 +0530, Atish Kathpal wrote:
> Hello friends
> 
> 
> I am running the "mkcephfs -a -c ceph.conf -k ceph.keyring" script as
> root (sudo) and have also tried by giving 777 privileges to all
> directories being accessed.
> 
> 
> I also made the suggested change in the ceph.conf file, by replacing
> with $id at required places. (following convention).
> 
> 
> 
> Still I see the same output as posted before.
> 
> 
> 
> My /ceph_data directory is a normal linux directory created using
> mkdir and not a separate partition. Is that a problem?

What kind of filesystem is it? btrfs? ext4? If it's ext4, are xatrrs
enabled?

Wido

> 
> 
> atish@atish-virtual-machine:/etc/ceph$  mkcephfs -a -c ceph.conf -k
> ceph.keyring
> temp dir is /tmp/mkcephfs.NHCO3zMk87
> preparing monmap in /tmp/mkcephfs.NHCO3zMk87/monmap
> /usr/bin/monmaptool --create --clobber --add alpha 10.72.147.207:6789
> --print /tmp/mkcephfs.NHCO3zMk87/monmap
> /usr/bin/monmaptool: monmap file /tmp/mkcephfs.NHCO3zMk87/monmap
> /usr/bin/monmaptool: generated fsid
> 24b2d358-c94e-6abf-42bf-82657b64ea81
> epoch 1
> fsid 24b2d358-c94e-6abf-42bf-82657b64ea81
> last_changed 2011-09-15 17:11:10.867824
> created 2011-09-15 17:11:10.867824
> 0: 10.72.147.207:6789/0 mon.alpha
> /usr/bin/monmaptool: writing epoch 1
> to /tmp/mkcephfs.NHCO3zMk87/monmap (1 monitors)
> === osd.0 ===
>  ** WARNING: Ceph is still under development.  Any feedback can be
> directed  **
>  **          at ceph-devel@vger.kernel.org or
> http://ceph.newdream.net/.     **
> 2011-09-15 17:11:11.754143 b78c16e0 AdminSocketConfigObs::init:
> failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain
> socket to '/var/run/ceph/osd.0.asok': error 2: No such file or
> directory
> 2011-09-15 17:11:11.754176 b78c16e0 AdminSocketConfigObs: failed to
> start AdminSocket
> 2011-09-15 17:11:11.759168 b78c16e0 filestore(/ceph_data/osd.0)
> FileStore::mkfs: failed to open /ceph_data/osd.0/fsid: error 13:
> Permission denied
> 2011-09-15 17:11:11.759191 b78c16e0 OSD::mkfs: FileStore::mkfs failed
> with error -13
> 2011-09-15 17:11:11.760877 b78c16e0  ** ERROR: error creating empty
> object store in /ceph_data/osd.0: error 13: Permission denied
> failed: '/sbin/mkcephfs -d /tmp/mkcephfs.NHCO3zMk87 --init-daemon
> osd.0'
> 
> 
> Looking forward to your suggestions.
> 
> 
> Regards
> Atish
> 
> 
> PS: I am unable to mail from my gmail account to the
> entire ceph-devel@vger.kernel.org mailing list. I first posted the
> mail using
> "http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/3023"
> website. Any ideas for this ?
> 
> 
> On Thu, Sep 15, 2011 at 4:58 PM, Wido den Hollander <wido@widodh.nl>
> wrote:
>         Hi,
>         
>         As Jantje said, it seems you are not running as root.
>         
>         But I do have some recommendations for your config:
>         
>         On Thu, 2011-09-15 at 06:14 +0000, Atish Kathpal wrote:
>         > [global]
>         >         auth supported = cephx
>         >        keyring = /etc/ceph/ceph.keyring
>         >
>         > [mon]
>         >         mon data = /ceph_data/mon.0
>         >
>         
>         
>         This section is globally, so if you'd add more monitors, they
>         would all
>         get /ceph_data/mon.0 as data dir.
>         
>         I'd do:
>         
>         [mon]
>            mon data = /ceph_data/mon.$id
>         
>         > [mds]
>         >
>         > [osd]
>         >         osd data = /ceph_data/osd.0
>         >         osd journal = /ceph_data/osd.0.journal
>         >         osd journal size = 1000
>         
>         
>         Same goes here for your OSD, do:
>         
>          osd data = /ceph_data/osd.$id
>          osd journal = /ceph_data/osd.$id.journal
>         
>         >
>         > [mon.0]
>         >         host = atish-virtual-machine
>         >         mon addr = 10.72.147.207:6789
>         
>         
>         Monitors should not be named "0", "1", etc, but more like:
>         * mon.alpha
>         * mon.beta
>         * mon.charlie
>         
>         >
>         > [osd.0]
>         >         host = atish-virtual-machine
>         >
>         > [mds.0]
>         >         host = atish-virtual-machine
>         
>         
>         
>         And you have only one OSD, is that correct?
>         
>         Wido
>         
> 
> 

Re: mkcephfs giving problems

[Josh Durgin]

 >> failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain
 >> socket to '/var/run/ceph/osd.0.asok': error 2: No such file or
 >> directory

Do the /var/run/ceph and /ceph_data/osd.0 directories exist?