Unable to mount cephfs - can't read superblock

Unable to mount cephfs - can't read superblock

[Adam Nielsen]

Hi again,

I'm trying to mount my new Ceph volume on a remote PC using cephfs.  I've 
followed the quick start guide, but when I try to mount the filesystem I get this:

remote$ mount -t ceph 192.168.0.6:6789:/ /mnt/ceph/
mount: 192.168.0.6:6789:/: can't read superblock

remote$ dmesg | tail
[951382.981690] libceph: client4105 fsid aa447ff8-8270-491b-b59e-2735e852eaf5
[951382.983486] libceph: mon0 192.168.0.6:6789 session established

I'm not sure what the problem is.  The osd, mon and mds daemons are running on 
the Ceph host and no traffic is firewalled between the two machines.  I have 
disabled authentication in ceph.conf.  The only issue I can see is this:

cephhost$ ceph health
HEALTH_WARN 192 pgs degraded; 192 pgs stuck unclean

I'm not sure what that means as I have only used mkcephfs and then started the 
ceph service.  I haven't actually used it yet so I'm not sure where the 
unclean pgs are coming from.  The underlying filesystem is ext4 though.

I'm a bit stuck as to what to do next.  I can't see why the filesystem won't 
mount remotely, and the troubleshooting docs say unclean pgs are related to 
outages which doesn't seem possible on a newly created volume with only one 
osd/mon/mds instance.

Any pointers would be much appreciated!

Many thanks,
Adam.

Re: Unable to mount cephfs - can't read superblock

[Wido den Hollander]

On 02/09/2013 11:40 AM, Adam Nielsen wrote:
> Hi again,
>
> I'm trying to mount my new Ceph volume on a remote PC using cephfs.
> I've followed the quick start guide, but when I try to mount the
> filesystem I get this:
>
> remote$ mount -t ceph 192.168.0.6:6789:/ /mnt/ceph/
> mount: 192.168.0.6:6789:/: can't read superblock
>
> remote$ dmesg | tail
> [951382.981690] libceph: client4105 fsid
> aa447ff8-8270-491b-b59e-2735e852eaf5
> [951382.983486] libceph: mon0 192.168.0.6:6789 session established
>
> I'm not sure what the problem is.  The osd, mon and mds daemons are
> running on the Ceph host and no traffic is firewalled between the two
> machines.  I have disabled authentication in ceph.conf.  The only issue
> I can see is this:
>
> cephhost$ ceph health
> HEALTH_WARN 192 pgs degraded; 192 pgs stuck unclean
>

Could you show the output of "ceph -s"

Also, which version of Ceph are you using under which OS?

Wido

> I'm not sure what that means as I have only used mkcephfs and then
> started the ceph service.  I haven't actually used it yet so I'm not
> sure where the unclean pgs are coming from.  The underlying filesystem
> is ext4 though.
>
> I'm a bit stuck as to what to do next.  I can't see why the filesystem
> won't mount remotely, and the troubleshooting docs say unclean pgs are
> related to outages which doesn't seem possible on a newly created volume
> with only one osd/mon/mds instance.
>
> Any pointers would be much appreciated!
>
> Many thanks,
> Adam.
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
Wido den Hollander
42on B.V.

Phone: +31 (0)20 700 9902
Skype: contact42on

Re: Unable to mount cephfs - can't read superblock

[Adam Nielsen]

Thanks for your quick reply!

> Could you show the output of "ceph -s"

$ ceph -s
    health HEALTH_WARN 192 pgs degraded; 192 pgs stuck unclean
    monmap e1: 1 mons at {0=192.168.0.6:6789/0}, election epoch 0, quorum 0 0
    osdmap e3: 1 osds: 1 up, 1 in
     pgmap v119: 192 pgs: 192 active+degraded; 0 bytes data, 10204 MB used, 
2740 GB / 2750 GB avail
    mdsmap e1: 0/0/1 up

> Also, which version of Ceph are you using under which OS?

The latest stable Debian release from ceph.com (bobtail AFAIK).

Thanks,
Adam.

Re: Unable to mount cephfs - can't read superblock

[Wido den Hollander]

On 02/09/2013 12:06 PM, Adam Nielsen wrote:
> Thanks for your quick reply!
>
>> Could you show the output of "ceph -s"
>
> $ ceph -s
>     health HEALTH_WARN 192 pgs degraded; 192 pgs stuck unclean
>     monmap e1: 1 mons at {0=192.168.0.6:6789/0}, election epoch 0,
> quorum 0 0
>     osdmap e3: 1 osds: 1 up, 1 in
>      pgmap v119: 192 pgs: 192 active+degraded; 0 bytes data, 10204 MB
> used, 2740 GB / 2750 GB avail
>     mdsmap e1: 0/0/1 up
>

Ah, I see you only have one OSD, where the default replication level is 
2. Also, pools don't work by default if only one replica is left.

You better add a second OSD or just do a mkcephfs again with a second 
OSD in the configuration.

Just a reminder, it's also in the docs, but CephFS is still in beta, so 
expect weird things to happen. It can however not hurt to play with it!

P.S.: There's also a new and shiny ceph-users list since two days ago, 
might want to subscribe there.

Wido

>> Also, which version of Ceph are you using under which OS?
>
> The latest stable Debian release from ceph.com (bobtail AFAIK).
>
> Thanks,
> Adam.
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
Wido den Hollander
42on B.V.

Phone: +31 (0)20 700 9902
Skype: contact42on

Re: Unable to mount cephfs - can't read superblock

[Adam Nielsen]

> Ah, I see you only have one OSD, where the default replication level is 2.
> Also, pools don't work by default if only one replica is left.
>
> You better add a second OSD or just do a mkcephfs again with a second OSD in
> the configuration.

Ah ok.  From my earlier post I think I can add the second OSD on the same disk 
since it is mounted on /var/lib/ceph/.  Are there likely to be any problems 
with this, going into production?  (I mean having two OSDs on the same node.) 
  I will eventually have a second node, but I know only having two isn't ideal.

> Just a reminder, it's also in the docs, but CephFS is still in beta, so expect
> weird things to happen. It can however not hurt to play with it!

Thanks for the reminder - but I only plan to use CephFS to store backup copies 
of things until it becomes stable, so hopefully I can give it some testing and 
survive if it breaks :-)

> P.S.: There's also a new and shiny ceph-users list since two days ago, might
> want to subscribe there.

I will use that list as soon as it appears on GMane, since I find their NNTP 
interface a lot easier than managing a bunch of mailing list subscriptions! 
Maybe someone with more authority than myself can add it?

   http://gmane.org/subscribe.php

Cheers,
Adam.

Re: Unable to mount cephfs - can't read superblock

[Ross David Turk]

On Feb 9, 2013, at 3:25 AM, Adam Nielsen <a.nielsen@shikadi.net> wrote:

> I will use that list as soon as it appears on GMane, since I find their NNTP interface a lot easier than managing a bunch of mailing list subscriptions! Maybe someone with more authority than myself can add it?
> 
>  http://gmane.org/subscribe.php

Agree - requested it last week, will follow up when it's added.

Cheers,
Ross

GMane for ceph-users (was Re: Unable to mount cephfs - can't read superblock)

[Ross David Turk]

On Feb 11, 2013, at 2:47 PM, Ross David Turk <ross@inktank.com> wrote:
> On Feb 9, 2013, at 3:25 AM, Adam Nielsen <a.nielsen@shikadi.net> wrote:
> 
>> I will use that list as soon as it appears on GMane, since I find their NNTP interface a lot easier than managing a bunch of mailing list subscriptions! Maybe someone with more authority than myself can add it?
>> 
>> http://gmane.org/subscribe.php
> 
> Agree - requested it last week, will follow up when it's added.

Hi!

GMane has now added ceph-users:
http://dir.gmane.org/gmane.comp.file-systems.ceph.user

I updated the Mailing List page on ceph.com to point to these archives instead so that people know they're there :)

Cheers,
Ross


--
Ross Turk
Community, Inktank

@rossturk @inktank @ceph

Re: Unable to mount cephfs - can't read superblock

[Gregory Farnum]

On Saturday, February 9, 2013 at 3:09 AM, Wido den Hollander wrote:
> On 02/09/2013 12:06 PM, Adam Nielsen wrote:
> > Thanks for your quick reply!
> > 
> > > Could you show the output of "ceph -s"
> > 
> > $ ceph -s
> > health HEALTH_WARN 192 pgs degraded; 192 pgs stuck unclean
> > monmap e1: 1 mons at {0=192.168.0.6:6789/0}, election epoch 0,
> > quorum 0 0
> > osdmap e3: 1 osds: 1 up, 1 in
> > pgmap v119: 192 pgs: 192 active+degraded; 0 bytes data, 10204 MB
> > used, 2740 GB / 2750 GB avail
> > mdsmap e1: 0/0/1 up
> 
> 
> 
> Ah, I see you only have one OSD, where the default replication level is 
> 2. Also, pools don't work by default if only one replica is left.

Actually that shouldn't be a problem with a size 2 pool. They'll work as long as at least half of their assigned replica count is available. :) (There was a brief period in development releases where that wasn't the case, but it didn't last very long!)

In any case, this output indicates that your MDS isn't actually running, Adam, or at least isn't connected. Check and see if the process is still going?
You should also have minimal logging by default in /var/lib/ceph/mds*; you might find some output there that could be useful.
-Greg

 
> 
> You better add a second OSD or just do a mkcephfs again with a second 
> OSD in the configuration.
> 
> Just a reminder, it's also in the docs, but CephFS is still in beta, so 
> expect weird things to happen. It can however not hurt to play with it!
> 
> P.S.: There's also a new and shiny ceph-users list since two days ago, 
> might want to subscribe there.
> 
> Wido
> 
> > > Also, which version of Ceph are you using under which OS?
> > 
> > The latest stable Debian release from ceph.com (http://ceph.com) (bobtail AFAIK).
> > 
> > Thanks,
> > Adam.
> > 
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org (mailto:majordomo@vger.kernel.org)
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> 
> 
> 
> 
> -- 
> Wido den Hollander
> 42on B.V.
> 
> Phone: +31 (0)20 700 9902
> Skype: contact42on
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org (mailto:majordomo@vger.kernel.org)
> More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Unable to mount cephfs - can't read superblock

[Adam Nielsen]

>>> $ ceph -s
>>> health HEALTH_WARN 192 pgs degraded; 192 pgs stuck unclean
>>> monmap e1: 1 mons at {0=192.168.0.6:6789/0}, election epoch 0,
>>> quorum 0 0
>>> osdmap e3: 1 osds: 1 up, 1 in
>>> pgmap v119: 192 pgs: 192 active+degraded; 0 bytes data, 10204 MB
>>> used, 2740 GB / 2750 GB avail
>>> mdsmap e1: 0/0/1 up
>>
> In any case, this output indicates that your MDS isn't actually running, Adam, or at least isn't connected. Check and see if the process is still going?
> You should also have minimal logging by default in /var/lib/ceph/mds*; you might find some output there that could be useful.

The MDS appears to be running:

$ ps -A | grep ceph
12903 ?        00:00:17 ceph-mon
12966 ?        00:00:10 ceph-mds
13047 ?        00:00:31 ceph-osd

And I found some logs in /var/log/ceph:

$ cat /var/log/ceph/ceph-mds.0.log 
2013-02-10 07:57:16.505842 b4aa3b70  0 mds.-1.0 ms_handle_connect on 192.168.0.6:6789/0

So it appears the mds is running.  Wireshark shows some traffic going between hosts when the mount request comes through, but then the responses stop and the client eventually gives up and the mount fails.

>> You better add a second OSD or just do a mkcephfs again with a second
>> OSD in the configuration.

I just tried this and it fixed the unclean pgs issue, but I still can't mount a cephfs filesystem:

$ ceph -s
   health HEALTH_OK
   monmap e1: 1 mons at {0=192.168.0.6:6789/0}, election epoch 0, quorum 0 0
   osdmap e5: 2 osds: 2 up, 2 in
    pgmap v107: 384 pgs: 384 active+clean; 0 bytes data, 40423 MB used, 5461 GB / 5501 GB avail
   mdsmap e1: 0/0/1 up

remote$ mount -t ceph 192.168.0.6:6789:/ /mnt/ceph/
mount: 192.168.0.6:6789:/: can't read superblock

Running the mds daemon in debug mode says this:

...
2013-02-10 08:07:03.550977 b2a83b70 10 mds.-1.0 MDS::ms_get_authorizer type=mon
2013-02-10 08:07:03.551840 b4a87b70  0 mds.-1.0 ms_handle_connect on 192.168.0.6:6789/0
2013-02-10 08:07:03.555307 b738c710 10 mds.-1.0 beacon_send up:boot seq 1 (currently up:boot)
2013-02-10 08:07:03.555629 b738c710 10 mds.-1.0 create_logger
2013-02-10 08:07:03.564138 b4a87b70  5 mds.-1.0 handle_mds_map epoch 1 from mon.0
2013-02-10 08:07:03.564348 b4a87b70 10 mds.-1.0      my compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object}
2013-02-10 08:07:03.564454 b4a87b70 10 mds.-1.0  mdsmap compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object}
2013-02-10 08:07:03.564547 b4a87b70 10 mds.-1.-1 map says i am 192.168.0.6:6800/16077 mds.-1.-1 state down:dne
2013-02-10 08:07:03.564654 b4a87b70 10 mds.-1.-1 not in map yet
2013-02-10 08:07:07.555567 b2881b70 10 mds.-1.-1 beacon_send up:boot seq 2 (currently down:dne)
2013-02-10 08:07:11.555858 b2881b70 10 mds.-1.-1 beacon_send up:boot seq 3 (currently down:dne)
2013-02-10 08:07:15.556123 b2881b70 10 mds.-1.-1 beacon_send up:boot seq 4 (currently down:dne)
2013-02-10 08:07:19.556411 b2881b70 10 mds.-1.-1 beacon_send up:boot seq 5 (currently down:dne)
2013-02-10 08:07:23.556654 b2881b70 10 mds.-1.-1 beacon_send up:boot seq 6 (currently down:dne)
2013-02-10 08:07:27.556931 b2881b70 10 mds.-1.-1 beacon_send up:boot seq 7 (currently down:dne)
2013-02-10 08:07:31.557189 b2881b70 10 mds.-1.-1 beacon_send up:boot seq 8 (currently down:dne)
...

I'm guessing there's something going wrong but I'm not sure what to do next!

Many thanks,
Adam.

Re: Unable to mount cephfs - can't read superblock

[Gregory Farnum]

On Sat, Feb 9, 2013 at 2:13 PM, Adam Nielsen <a.nielsen@shikadi.net> wrote:
>>>> $ ceph -s
>>>> health HEALTH_WARN 192 pgs degraded; 192 pgs stuck unclean
>>>> monmap e1: 1 mons at {0=192.168.0.6:6789/0}, election epoch 0,
>>>> quorum 0 0
>>>> osdmap e3: 1 osds: 1 up, 1 in
>>>> pgmap v119: 192 pgs: 192 active+degraded; 0 bytes data, 10204 MB
>>>> used, 2740 GB / 2750 GB avail
>>>> mdsmap e1: 0/0/1 up
>>>
>> In any case, this output indicates that your MDS isn't actually running, Adam, or at least isn't connected. Check and see if the process is still going?
>> You should also have minimal logging by default in /var/lib/ceph/mds*; you might find some output there that could be useful.
>
> The MDS appears to be running:
>
> $ ps -A | grep ceph
> 12903 ?        00:00:17 ceph-mon
> 12966 ?        00:00:10 ceph-mds
> 13047 ?        00:00:31 ceph-osd
>
> And I found some logs in /var/log/ceph:
>
> $ cat /var/log/ceph/ceph-mds.0.log
> 2013-02-10 07:57:16.505842 b4aa3b70  0 mds.-1.0 ms_handle_connect on 192.168.0.6:6789/0
>
> So it appears the mds is running.  Wireshark shows some traffic going between hosts when the mount request comes through, but then the responses stop and the client eventually gives up and the mount fails.
>
>>> You better add a second OSD or just do a mkcephfs again with a second
>>> OSD in the configuration.
>
> I just tried this and it fixed the unclean pgs issue, but I still can't mount a cephfs filesystem:
>
> $ ceph -s
>    health HEALTH_OK
>    monmap e1: 1 mons at {0=192.168.0.6:6789/0}, election epoch 0, quorum 0 0
>    osdmap e5: 2 osds: 2 up, 2 in
>     pgmap v107: 384 pgs: 384 active+clean; 0 bytes data, 40423 MB used, 5461 GB / 5501 GB avail
>    mdsmap e1: 0/0/1 up
>
> remote$ mount -t ceph 192.168.0.6:6789:/ /mnt/ceph/
> mount: 192.168.0.6:6789:/: can't read superblock
>
> Running the mds daemon in debug mode says this:
>
> ...
> 2013-02-10 08:07:03.550977 b2a83b70 10 mds.-1.0 MDS::ms_get_authorizer type=mon
> 2013-02-10 08:07:03.551840 b4a87b70  0 mds.-1.0 ms_handle_connect on 192.168.0.6:6789/0
> 2013-02-10 08:07:03.555307 b738c710 10 mds.-1.0 beacon_send up:boot seq 1 (currently up:boot)
> 2013-02-10 08:07:03.555629 b738c710 10 mds.-1.0 create_logger
> 2013-02-10 08:07:03.564138 b4a87b70  5 mds.-1.0 handle_mds_map epoch 1 from mon.0
> 2013-02-10 08:07:03.564348 b4a87b70 10 mds.-1.0      my compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object}
> 2013-02-10 08:07:03.564454 b4a87b70 10 mds.-1.0  mdsmap compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object}
> 2013-02-10 08:07:03.564547 b4a87b70 10 mds.-1.-1 map says i am 192.168.0.6:6800/16077 mds.-1.-1 state down:dne
> 2013-02-10 08:07:03.564654 b4a87b70 10 mds.-1.-1 not in map yet
> 2013-02-10 08:07:07.555567 b2881b70 10 mds.-1.-1 beacon_send up:boot seq 2 (currently down:dne)
> 2013-02-10 08:07:11.555858 b2881b70 10 mds.-1.-1 beacon_send up:boot seq 3 (currently down:dne)
> 2013-02-10 08:07:15.556123 b2881b70 10 mds.-1.-1 beacon_send up:boot seq 4 (currently down:dne)
> 2013-02-10 08:07:19.556411 b2881b70 10 mds.-1.-1 beacon_send up:boot seq 5 (currently down:dne)
> 2013-02-10 08:07:23.556654 b2881b70 10 mds.-1.-1 beacon_send up:boot seq 6 (currently down:dne)
> 2013-02-10 08:07:27.556931 b2881b70 10 mds.-1.-1 beacon_send up:boot seq 7 (currently down:dne)
> 2013-02-10 08:07:31.557189 b2881b70 10 mds.-1.-1 beacon_send up:boot seq 8 (currently down:dne)
> ...

How bizarre. That indicates the MDS is running and is requesting to
become active, but the monitor for some reason isn't letting it in.
Can you restart your monitor with logging on as well (--debug_mon 20
on the end of the command line, or "debug mon = 20" in the config) and
then try again?
The other possibility is that maybe your MDS doesn't have the right
access permissions; does "ceph auth list" include an MDS, and does it
have any permissions associated?
-Greg