mount: 10.0.6.10:/: can't read superblock

mount: 10.0.6.10:/: can't read superblock

[Martin Wilderoth]

Hello

Is there a way to recover this error.

mount -t ceph 10.0.6.10:/ /mnt -vv -o name=admin,secret=XXXXXXXXXXXXXXXXXXXXXXX
[  506.640433] libceph: loaded (mon/osd proto 15/24, osdmap 5/6 5/6)
[  506.650594] ceph: loaded (mds proto 32)
[  506.652353] libceph: client0 fsid a9d5f9e1-4bb9-4fab-b79b-ba4457631b01
[  506.670876] Intel AES-NI instructions are not detected.
[  506.678861] libceph: mon0 10.0.6.10:6789 session established
mount: 10.0.6.10:/: can't read superblock

One of my mds logs has 24G of data. 

I have some rbd devices that I would like to keep.

/Regards Martin

Re: mount: 10.0.6.10:/: can't read superblock

[Wido den Hollander]

Hi Martin,

On 06/05/2012 08:07 PM, Martin Wilderoth wrote:
> Hello
>
> Is there a way to recover this error.
>
> mount -t ceph 10.0.6.10:/ /mnt -vv -o name=admin,secret=XXXXXXXXXXXXXXXXXXXXXXX
> [  506.640433] libceph: loaded (mon/osd proto 15/24, osdmap 5/6 5/6)
> [  506.650594] ceph: loaded (mds proto 32)
> [  506.652353] libceph: client0 fsid a9d5f9e1-4bb9-4fab-b79b-ba4457631b01
> [  506.670876] Intel AES-NI instructions are not detected.
> [  506.678861] libceph: mon0 10.0.6.10:6789 session established
> mount: 10.0.6.10:/: can't read superblock
>

Could you share some more information? For example the output from: ceph -s

Did you change anything to the cluster since it worked? And what version 
are you running?

> One of my mds logs has 24G of data.

Is it still running?

>
> I have some rbd devices that I would like to keep.

RBD doesn't use the MDS nor the POSIX filesystem, so you will probably 
be fine, but we need the output of "ceph -s" first.

Does this work?
$ rbd ls
$ rados -p rbd ls

Wido

>
> /Regards Martin
>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mount: 10.0.6.10:/: can't read superblock

[Martin Wilderoth]

Hello

> Hi Martin, 
>
> On 06/05/2012 08:07 PM, Martin Wilderoth wrote: 
> > Hello 
> >
> > Is there a way to recover this error. 
> >
> > mount -t ceph 10.0.6.10:/ /mnt -vv -o name=admin,secret=XXXXXXXXXXXXXXXXXXXXXXX 
> > [ 506.640433] libceph: loaded (mon/osd proto 15/24, osdmap 5/6 5/6) 
> > [ 506.650594] ceph: loaded (mds proto 32) 
> > [ 506.652353] libceph: client0 fsid a9d5f9e1-4bb9-4fab-b79b-ba4457631b01 
> > [ 506.670876] Intel AES-NI instructions are not detected. 
> > [ 506.678861] libceph: mon0 10.0.6.10:6789 session established 
> > mount: 10.0.6.10:/: can't read superblock 
> >
>
> Could you share some more information? For example the output from: ceph -s 

2012-06-05 20:25:05.307914    pg v1189604: 1152 pgs: 1152 active+clean; 191 GB data, 393 GB used, 973 GB / 1379 GB avail
2012-06-05 20:25:05.315871   mds e60: 1/1/1 up {0=c=up:replay}, 2 up:standby
2012-06-05 20:25:05.315965   osd e1106: 8 osds: 8 up, 8 in
2012-06-05 20:25:05.316165   log 2012-06-05 20:24:50.425527 mon.0 10.0.6.10:6789/0 75 : [INF] mds.? 10.0.6.11:6800/22974 up:boot
2012-06-05 20:25:05.316371   mon e1: 3 mons at {a=10.0.6.10:6789/0,b=10.0.6.11:6789/0,c=10.0.6.12:6789/0}


>
> Did you change anything to the cluster since it worked? And what version 
> are you running? 

I have not done any changes installed at version 0.46 upgraded earlier and have been testing with
ceph and ceph-fuse and backuppc. It was during the ceph-fuse it hanged.

Current version
ceph version 0.47.2 (commit:8bf9fde89bd6ebc4b0645b2fe02dadb1c17ad372)

> > One of my mds logs has 24G of data. 
>
> Is it still running? 
I have restarted mds.a and mds.b they seems to be running. But not everything.
mds.a was stoped not sure mds.b but it has a big logfile.

>
> > 
> > I have some rbd devices that I would like to keep. 
>
> RBD doesn't use the MDS nor the POSIX filesystem, so you will probably 
> be fine, but we need the output of "ceph -s" first. 
>
> Does this work? 
> $ rbd ls
this works I'm still using the rbd with no problem 
> $ rados -p rbd ls 
seems to work reports something simmilar to
rb.0.2.00000000052e
rb.0.0.0000000002f2
rb.0.7.000000000345
rb.0.7.000000000896
rb.0.0.000000000102
rb.0.9.000000000172
rb.0.1.000000000350
rb.0.4.000000000180
rb.0.4.00000000068b
rb.0.5.00000000054c
rb.0.2.0000000001e1
 
> Wido 
>
> > 
> > /Regards Martin 
> > 

Re: mount: 10.0.6.10:/: can't read superblock

[Martin Wilderoth]

Hello Again,

I restarted the mds on all servers and then it worked again

 /Regards Martin

> Hello 
> 
> > Hi Martin, 
> > 
> > On 06/05/2012 08:07 PM, Martin Wilderoth wrote: 
> > > Hello 
> > > 
> > > Is there a way to recover this error. 
> > > 
> > > mount -t ceph 10.0.6.10:/ /mnt -vv -o name=admin,secret=XXXXXXXXXXXXXXXXXXXXXXX 
> > > [ 506.640433] libceph: loaded (mon/osd proto 15/24, osdmap 5/6 5/6) 
> > > [ 506.650594] ceph: loaded (mds proto 32) 
> > > [ 506.652353] libceph: client0 fsid a9d5f9e1-4bb9-4fab-b79b-ba4457631b01 
> > > [ 506.670876] Intel AES-NI instructions are not detected. 
> > > [ 506.678861] libceph: mon0 10.0.6.10:6789 session established 
> > > mount: 10.0.6.10:/: can't read superblock 
> > > 
> > 
> > Could you share some more information? For example the output from: ceph -s 
> 
> 2012-06-05 20:25:05.307914 pg v1189604: 1152 pgs: 1152 active+clean; 191 GB data, 393 GB used, 973 GB / 1379 GB avail 
> 012-06-05 20:25:05.315871 mds e60: 1/1/1 up {0=c=up:replay}, 2 up:standby 
> 2012-06-05 20:25:05.315965 osd e1106: 8 osds: 8 up, 8 in 
> 2012-06-05 20:25:05.316165 log 2012-06-05 20:24:50.425527 mon.0 10.0.6.10:6789/0 75 : [INF] mds.? 10.0.6.11:6800/22974 up:boot 
> 2012-06-05 20:25:05.316371 mon e1: 3 mons at {a=10.0.6.10:6789/0,b=10.0.6.11:6789/0,c=10.0.6.12:6789/0} 
> 
> 
> > 
> > Did you change anything to the cluster since it worked? And what version 
> > are you running? 
> 
> I have not done any changes installed at version 0.46 upgraded earlier and have been testing with 
> ceph and ceph-fuse and backuppc. It was during the ceph-fuse it hanged. 
> 
> Current version 
> ceph version 0.47.2 (commit:8bf9fde89bd6ebc4b0645b2fe02dadb1c17ad372) 
> 
> > > One of my mds logs has 24G of data. 
> > 
> > Is it still running? 
> I have restarted mds.a and mds.b they seems to be running. But not everything. 
> mds.a was stoped not sure mds.b but it has a big logfile. 
> 
> > 
> > > 
> > > I have some rbd devices that I would like to keep. 
> > 
> > RBD doesn't use the MDS nor the POSIX filesystem, so you will probably 
> > be fine, but we need the output of "ceph -s" first. 
> > 
> > Does this work? 
> > $ rbd ls 
> this works I'm still using the rbd with no problem 
> > $ rados -p rbd ls 
> seems to work reports something simmilar to 
> rb.0.2.00000000052e 
> rb.0.0.0000000002f2 
> rb.0.7.000000000345 
> rb.0.7.000000000896 
> rb.0.0.000000000102 
> rb.0.9.000000000172 
> rb.0.1.000000000350 
> rb.0.4.000000000180 
> rb.0.4.00000000068b 
> rb.0.5.00000000054c 
> rb.0.2.0000000001e1 
> 
> > Wido 
> > 
> > > 
> > > /Regards Martin 
> > > 

Re: mount: 10.0.6.10:/: can't read superblock

[Greg Farnum]

For future reference, that error was because the active MDS server was in replay. I can't tell why it didn't move on to active from what you posted, but I imagine it just got a little stuck since restarting made it work out. 
-Greg


On Tuesday, June 5, 2012 at 1:05 PM, Martin Wilderoth wrote:

> Hello Again,
> 
> I restarted the mds on all servers and then it worked again
> 
> /Regards Martin
> 
> > Hello 
> > 
> > > Hi Martin, 
> > > 
> > > On 06/05/2012 08:07 PM, Martin Wilderoth wrote: 
> > > > Hello 
> > > > 
> > > > Is there a way to recover this error. 
> > > > 
> > > > mount -t ceph 10.0.6.10:/ /mnt -vv -o name=admin,secret=XXXXXXXXXXXXXXXXXXXXXXX 
> > > > [ 506.640433] libceph: loaded (mon/osd proto 15/24, osdmap 5/6 5/6) 
> > > > [ 506.650594] ceph: loaded (mds proto 32) 
> > > > [ 506.652353] libceph: client0 fsid a9d5f9e1-4bb9-4fab-b79b-ba4457631b01 
> > > > [ 506.670876] Intel AES-NI instructions are not detected. 
> > > > [ 506.678861] libceph: mon0 10.0.6.10:6789 session established 
> > > > mount: 10.0.6.10:/: can't read superblock 
> > > 
> > > 
> > > 
> > > Could you share some more information? For example the output from: ceph -s 
> > 
> > 2012-06-05 20:25:05.307914 pg v1189604: 1152 pgs: 1152 active+clean; 191 GB data, 393 GB used, 973 GB / 1379 GB avail 
> > 012-06-05 20:25:05.315871 mds e60: 1/1/1 up {0=c=up:replay}, 2 up:standby 
> > 2012-06-05 20:25:05.315965 osd e1106: 8 osds: 8 up, 8 in 
> > 2012-06-05 20:25:05.316165 log 2012-06-05 20:24:50.425527 mon.0 10.0.6.10:6789/0 75 : [INF] mds.? 10.0.6.11:6800/22974 up:boot 
> > 2012-06-05 20:25:05.316371 mon e1: 3 mons at {a=10.0.6.10:6789/0,b=10.0.6.11:6789/0,c=10.0.6.12:6789/0} 
> > 
> > 
> > > 
> > > Did you change anything to the cluster since it worked? And what version 
> > > are you running? 
> > 
> > 
> > 
> > I have not done any changes installed at version 0.46 upgraded earlier and have been testing with 
> > ceph and ceph-fuse and backuppc. It was during the ceph-fuse it hanged. 
> > 
> > Current version 
> > ceph version 0.47.2 (commit:8bf9fde89bd6ebc4b0645b2fe02dadb1c17ad372) 
> > 
> > > > One of my mds logs has 24G of data. 
> > > 
> > > Is it still running? 
> > I have restarted mds.a and mds.b they seems to be running. But not everything. 
> > mds.a was stoped not sure mds.b but it has a big logfile. 
> > 
> > > 
> > > > 
> > > > I have some rbd devices that I would like to keep. 
> > > 
> > > RBD doesn't use the MDS nor the POSIX filesystem, so you will probably 
> > > be fine, but we need the output of "ceph -s" first. 
> > > 
> > > Does this work? 
> > > $ rbd ls 
> > 
> > 
> > this works I'm still using the rbd with no problem 
> > > $ rados -p rbd ls 
> > 
> > 
> > seems to work reports something simmilar to 
> > rb.0.2.00000000052e 
> > rb.0.0.0000000002f2 
> > rb.0.7.000000000345 
> > rb.0.7.000000000896 
> > rb.0.0.000000000102 
> > rb.0.9.000000000172 
> > rb.0.1.000000000350 
> > rb.0.4.000000000180 
> > rb.0.4.00000000068b 
> > rb.0.5.00000000054c 
> > rb.0.2.0000000001e1 
> > 
> > > Wido 
> > > 
> > > > 
> > > > /Regards Martin 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org (mailto:majordomo@vger.kernel.org)
> More majordomo info at http://vger.kernel.org/majordomo-info.html

Ceph and BackupPC (was: Re: mount: 10.0.6.10:/: can't read superblock)

[Guido Winkelmann]

Am Dienstag, 5. Juni 2012, 14:33:19 schrieb Martin Wilderoth:

> I have not done any changes installed at version 0.46 upgraded earlier and
> have been testing with ceph and ceph-fuse and backuppc.

BackupPC and Ceph sounds like an interesting combination. Other than this 
hanging MDS problem, what are your experiences with that?

I'm asking because BackupPC is the only application I know of that actually 
makes heavy use of hardlinks. Ceph, as far as I've understood it, supports 
hardlinks, but works under the assumption that they're a rare and not 
performance-critical corner case, and does some optimizations that help the 
more common cases, but would probably hurt if you are using hardlinks a lot.

Regards,
	Guido

Re: Ceph and BackupPC (was: Re: mount: 10.0.6.10:/: can't read superblock)

[Sage Weil]

On Wed, 6 Jun 2012, Guido Winkelmann wrote:
> Am Dienstag, 5. Juni 2012, 14:33:19 schrieb Martin Wilderoth:
> 
> > I have not done any changes installed at version 0.46 upgraded earlier and
> > have been testing with ceph and ceph-fuse and backuppc.
> 
> BackupPC and Ceph sounds like an interesting combination. Other than this 
> hanging MDS problem, what are your experiences with that?
> 
> I'm asking because BackupPC is the only application I know of that actually 
> makes heavy use of hardlinks. Ceph, as far as I've understood it, supports 
> hardlinks, but works under the assumption that they're a rare and not 
> performance-critical corner case, and does some optimizations that help the 
> more common cases, but would probably hurt if you are using hardlinks a lot.

Right.  There is currently a scalability limit on the number of hard 
links, currently.  Any large set of backuppc data will eat a lot of memory 
on mds.0 and (eventually) case problems.

sage

Re: Ceph and BackupPC

[Jerker Nyberg]

On Wed, 6 Jun 2012, Guido Winkelmann wrote:

> I'm asking because BackupPC is the only application I know of that actually
> makes heavy use of hardlinks.

I used to run "rsync --link-dest" to create snapshot-like directories with 
hard links but have been using real ZFS snapshots lately.

--jerker

Re: mount: 10.0.6.10:/: can't read superblock

[Tommi Virtanen]

On Tue, Jun 5, 2012 at 9:37 PM, Martin Wilderoth
<martin.wilderoth@linserv.se> wrote:
> 0> 2012-06-06 05:38:37.200297 7f2d5ea08700 -1 mds/AnchorServer.cc: In function 'virtual void AnchorServer::handle_query(MMDSTableRequest*)' thread 7f2d5ea08700 time 2012-06-06 05:38:37.198981
> mds/AnchorServer.cc: 249: FAILED assert(anchor_map.count(curino) == 1)
>
> ceph version 0.46 (commit:cb7f1c9c7520848b0899b26440ac34a8acea58d1)
> 1: (AnchorServer::handle_query(MMDSTableRequest*)+0x175) [0x6bdc95]
> 2: (MDS::handle_deferrable_message(Message*)+0xd84) [0x4b0474]
> 3: (MDS::_dispatch(Message*)+0xaf8) [0x4c50b8]
...
> root@ceph1:~# ceph -v
> ceph version 0.47.2 (commit:8bf9fde89bd6ebc4b0645b2fe02dadb1c17ad372)
> root@ceph1:~# ssh ceph2 ceph -v
> ceph version 0.47.2 (commit:8bf9fde89bd6ebc4b0645b2fe02dadb1c17ad372)
> root@ceph1:~# ssh ceph3 ceph -v
> ceph version 0.47.2 (commit:8bf9fde89bd6ebc4b0645b2fe02dadb1c17ad372)
> root@ceph1:~# ssh ceph4 ceph -v
> ceph version 0.47.2 (commit:8bf9fde89bd6ebc4b0645b2fe02dadb1c17ad372)
>
> is the 0.46 above reporting when the error occurred or am I running the wrong binaries
> i use the debian packages ?

That sounds weird. The ways I can see that being possible are

1. you had daemons still running from when you had 0.46 debs
installed, and the package upgrade didn't restart them -- ceph.git
uses "dh_installinit --no-start", so this might actually have
happened?

2. at some point, you checked out the 0.46 source and ran make
install, and the binaries from that are installed at a path that takes
precedence over the 0.47.2 ones in /usr/sbin

But having mixed versions in your cluster might explain the crashes.

Re: mount: 10.0.6.10:/: can't read superblock

[Martin Wilderoth]

Hello

> On Tue, Jun 5, 2012 at 9:37 PM, Martin Wilderoth 
> <martin.wilderoth@linserv.se> wrote: 
> > 0> 2012-06-06 05:38:37.200297 7f2d5ea08700 -1 mds/AnchorServer.cc: In function 'virtual void >AnchorServer::handle_query(MMDSTableRequest*)' thread 7f2d5ea08700 time 2012-06-06 05:38:37.198981 
> > mds/AnchorServer.cc: 249: FAILED assert(anchor_map.count(curino) == 1) 
> > 
> > ceph version 0.46 (commit:cb7f1c9c7520848b0899b26440ac34a8acea58d1) 
> > 1: (AnchorServer::handle_query(MMDSTableRequest*)+0x175) [0x6bdc95] 
> > 2: (MDS::handle_deferrable_message(Message*)+0xd84) [0x4b0474] 
> > 3: (MDS::_dispatch(Message*)+0xaf8) [0x4c50b8] 
... 
> > root@ceph1:~# ceph -v 
> > ceph version 0.47.2 (commit:8bf9fde89bd6ebc4b0645b2fe02dadb1c17ad372) 
> > root@ceph1:~# ssh ceph2 ceph -v 
> > ceph version 0.47.2 (commit:8bf9fde89bd6ebc4b0645b2fe02dadb1c17ad372) 
> > root@ceph1:~# ssh ceph3 ceph -v 
> > ceph version 0.47.2 (commit:8bf9fde89bd6ebc4b0645b2fe02dadb1c17ad372) 
> > root@ceph1:~# ssh ceph4 ceph -v 
> > ceph version 0.47.2 (commit:8bf9fde89bd6ebc4b0645b2fe02dadb1c17ad372) 
> > 
> > is the 0.46 above reporting when the error occurred or am I running the wrong binaries 
> > i use the debian packages ? 
> 
> That sounds weird. The ways I can see that being possible are 
> 
> 1. you had daemons still running from when you had 0.46 debs 
> installed, and the package upgrade didn't restart them -- ceph.git 
> uses "dh_installinit --no-start", so this might actually have 
> happened? 

When upgrading from different versions I just run apt-get upgrade in a running cluster.
I just stop ods,mds and mon on the host and then upgrade.

Maybe this is very bad ? 

After the update I restart the server and take the next one.
But it means that I'm running nodes with different version.

In the last update I got HEALTH problems during the upgrade but when they were all upgraded
it looked ok.

The reason for doing like this is that if during the upgrade the upgrade host fails it's better
to have the other tree running to be able to recover.

But maybe this uppgrading process could create crashes :-(.

Maybe better to stop the cluster and restart when upgraded. 
But then the ceph.conf file has to be removed before restarting the host and copied back
before starting it again.

> 
> 2. at some point, you checked out the 0.46 source and ran make 
> install, and the binaries from that are installed at a path that takes 
> precedence over the 0.47.2 ones in /usr/sbin 
> 
> But having mixed versions in your cluster might explain the crashes. 

It was one package that was kept back during the upgrade. So I was runnig the wrong binary
on the mds maybe more ???. The package name was ceph, not sure what binaries that were missing.
but ceph -v reported 0.47.2. But doing apt-get install ceph installed the missing binary and
one other package.

 /Best Regards Martin

Re: mount: 10.0.6.10:/: can't read superblock

[Tommi Virtanen]

On Fri, Jun 8, 2012 at 12:44 PM, Martin Wilderoth
<martin.wilderoth@linserv.se> wrote:
> When upgrading from different versions I just run apt-get upgrade in a running cluster.
> I just stop ods,mds and mon on the host and then upgrade.
>
> Maybe this is very bad ?

In general, we work to make upgrades safe. MDS may not be getting as
much attention as the other daemons, so that might be the issue here.