OSD::mkfs: couldn't mount FileStore: error -22

OSD::mkfs: couldn't mount FileStore: error -22

[Adam Nielsen]

Hi all,

I'm attempting to set up a Ceph cluster for testing, initially with only one 
node so I can add others next.  I think I've followed the guides as closely as 
I could, but now when running mkcephfs it fails and I can't find much about 
the error through Google.

What am I doing wrong?  This is on Debian Squeeze with the stable repo from 
ceph.com.

$ mkcephfs -a -c /etc/ceph/ceph.conf
temp dir is /tmp/mkcephfs.FqyH7RCFY1
preparing monmap in /tmp/mkcephfs.FqyH7RCFY1/monmap
/usr/bin/monmaptool --create --clobber --add 0 192.168.0.6:6789 --print 
/tmp/mkcephfs.FqyH7RCFY1/monmap
/usr/bin/monmaptool: monmap file /tmp/mkcephfs.FqyH7RCFY1/monmap
/usr/bin/monmaptool: generated fsid 77b0e551-b545-42df-957e-b99292539501
epoch 0
fsid 77b0e551-b545-42df-957e-b99292539501
last_changed 2012-10-14 17:40:11.128447
created 2012-10-14 17:40:11.128447
0: 192.168.0.6:6789/0 mon.0
/usr/bin/monmaptool: writing epoch 0 to /tmp/mkcephfs.FqyH7RCFY1/monmap (1 
monitors)
=== osd.0 ===
  ** WARNING: 'filestore btrfs snap' is enabled (for safe transactions,
              rollback), but btrfs does not support the SNAP_CREATE_V2 ioctl
              (added in Linux 2.6.37).  Expect slow btrfs sync/commit
              performance.
2012-10-14 17:40:12.768973 b732b710 -1 filestore(/var/lib/ceph/osd/ceph-0) 
mount initial op seq is 0; something is wrong
2012-10-14 17:40:12.769238 b732b710 -1 OSD::mkfs: couldn't mount FileStore: 
error -22
2012-10-14 17:40:12.769522 b732b710 -1  ** ERROR: error creating empty object 
store in /var/lib/ceph/osd/ceph-0: (22) Invalid argument
failed: '/sbin/mkcephfs -d /tmp/mkcephfs.FqyH7RCFY1 --init-daemon osd.0'

$ cat /etc/ceph/ceph.conf
[osd]
         osd journal size = 10000

[mon.0]
         host = node1
         mon addr = 192.168.0.6:6789
         mon data = /var/lib/ceph/osd/$cluster-$id/

[osd.0]
         host = node1

[mds.0]
         host = node1

Any suggestions?  I tried commenting out "mon data" but it didn't make a 
difference (and I need this in because the test machine's root filesystem is 
on a 4GB CF card, so I need all non-OS data to be stored on the disks assigned 
to Ceph.)

Thanks,
Adam.

Re: OSD::mkfs: couldn't mount FileStore: error -22

[Dan Mick]

The immediate cause of the problem is that the osd's "commit_op_seq" 
file is reading back '0',
which is invalid; it's created with an initial value of 1.  Try removing 
the osd data dir
(/var/lib/ceph/osd/ceph-0) completely and let it be recreated; perhaps 
something got there somehow by mistake.

If the error persists you can look more deeply.

On 10/14/2012 12:50 AM, Adam Nielsen wrote:
> Hi all,
>
> I'm attempting to set up a Ceph cluster for testing, initially with 
> only one node so I can add others next.  I think I've followed the 
> guides as closely as I could, but now when running mkcephfs it fails 
> and I can't find much about the error through Google.
>
> What am I doing wrong?  This is on Debian Squeeze with the stable repo 
> from ceph.com.
>
> $ mkcephfs -a -c /etc/ceph/ceph.conf
> temp dir is /tmp/mkcephfs.FqyH7RCFY1
> preparing monmap in /tmp/mkcephfs.FqyH7RCFY1/monmap
> /usr/bin/monmaptool --create --clobber --add 0 192.168.0.6:6789 
> --print /tmp/mkcephfs.FqyH7RCFY1/monmap
> /usr/bin/monmaptool: monmap file /tmp/mkcephfs.FqyH7RCFY1/monmap
> /usr/bin/monmaptool: generated fsid 77b0e551-b545-42df-957e-b99292539501
> epoch 0
> fsid 77b0e551-b545-42df-957e-b99292539501
> last_changed 2012-10-14 17:40:11.128447
> created 2012-10-14 17:40:11.128447
> 0: 192.168.0.6:6789/0 mon.0
> /usr/bin/monmaptool: writing epoch 0 to 
> /tmp/mkcephfs.FqyH7RCFY1/monmap (1 monitors)
> === osd.0 ===
>  ** WARNING: 'filestore btrfs snap' is enabled (for safe transactions,
>              rollback), but btrfs does not support the SNAP_CREATE_V2 
> ioctl
>              (added in Linux 2.6.37).  Expect slow btrfs sync/commit
>              performance.
> 2012-10-14 17:40:12.768973 b732b710 -1 
> filestore(/var/lib/ceph/osd/ceph-0) mount initial op seq is 0; 
> something is wrong
> 2012-10-14 17:40:12.769238 b732b710 -1 OSD::mkfs: couldn't mount 
> FileStore: error -22
> 2012-10-14 17:40:12.769522 b732b710 -1  ** ERROR: error creating empty 
> object store in /var/lib/ceph/osd/ceph-0: (22) Invalid argument
> failed: '/sbin/mkcephfs -d /tmp/mkcephfs.FqyH7RCFY1 --init-daemon osd.0'
>
> $ cat /etc/ceph/ceph.conf
> [osd]
>         osd journal size = 10000
>
> [mon.0]
>         host = node1
>         mon addr = 192.168.0.6:6789
>         mon data = /var/lib/ceph/osd/$cluster-$id/
>
> [osd.0]
>         host = node1
>
> [mds.0]
>         host = node1
>
> Any suggestions?  I tried commenting out "mon data" but it didn't make 
> a difference (and I need this in because the test machine's root 
> filesystem is on a 4GB CF card, so I need all non-OS data to be stored 
> on the disks assigned to Ceph.)
>
> Thanks,
> Adam.
>
> -- 
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OSD::mkfs: couldn't mount FileStore: error -22

[Adam Nielsen]

> The immediate cause of the problem is that the osd's "commit_op_seq" file is
> reading back '0',
> which is invalid; it's created with an initial value of 1.  Try removing the
> osd data dir
> (/var/lib/ceph/osd/ceph-0) completely and let it be recreated; perhaps
> something got there somehow by mistake.

Well it looks like commit_op_seq was never created properly:

$ ls /var/lib/ceph/osd/ceph-0/current/
total 4.0K
drwxr-xr-x 1 root root 26 Oct 14 17:40 .
drwxrwx--- 1 root root 74 Oct 14 17:40 ..
-rw-r--r-- 1 root root  0 Oct 14 17:40 commit_op_seq

But how do I delete it to start again?

$ rm -rf current/
rm: cannot remove `current': Directory not empty
$ ls current/
total 4.0K
drwxr-xr-x 1 root root  0 Oct 15 09:16 .
drwxrwx--- 1 root root 26 Oct 15 09:16 ..

This is on a brand new btrfs filesystem mounted in /var/lib/ceph/osd/ceph-0. 
I created it with this:

$ mkfs.btrfs -L data -s 4096 /dev/sda1

Did I do something wrong?

Thanks,
Adam.

Re: OSD::mkfs: couldn't mount FileStore: error -22

[Sage Weil]

On Mon, 15 Oct 2012, Adam Nielsen wrote:
> > The immediate cause of the problem is that the osd's "commit_op_seq" file is
> > reading back '0',
> > which is invalid; it's created with an initial value of 1.  Try removing the
> > osd data dir
> > (/var/lib/ceph/osd/ceph-0) completely and let it be recreated; perhaps
> > something got there somehow by mistake.
> 
> Well it looks like commit_op_seq was never created properly:
> 
> $ ls /var/lib/ceph/osd/ceph-0/current/
> total 4.0K
> drwxr-xr-x 1 root root 26 Oct 14 17:40 .
> drwxrwx--- 1 root root 74 Oct 14 17:40 ..
> -rw-r--r-- 1 root root  0 Oct 14 17:40 commit_op_seq
> 
> But how do I delete it to start again?
> 
> $ rm -rf current/
> rm: cannot remove `current': Directory not empty
> $ ls current/
> total 4.0K
> drwxr-xr-x 1 root root  0 Oct 15 09:16 .
> drwxrwx--- 1 root root 26 Oct 15 09:16 ..

current/ is a btrfs subvolume.. 'btrfs sub delete current' will remove it.

> This is on a brand new btrfs filesystem mounted in /var/lib/ceph/osd/ceph-0. I
> created it with this:
> 
> $ mkfs.btrfs -L data -s 4096 /dev/sda1
> 
> Did I do something wrong?

The warning in the previous email suggets you're running a fairly old 
kernel.. there is probably something handled incorrectly during the fs 
init process.  Exactly which kernel are you running?

In any case, btrfs isn't going to work particularly well on something that 
old; I suggest running something newer (3.5 or 3.6) or switching to XFS.

sage

Re: OSD::mkfs: couldn't mount FileStore: error -22

[Adam Nielsen]

> current/ is a btrfs subvolume.. 'btrfs sub delete current' will remove it.

Ah, that worked, thanks.  Unfortunately mkcephfs still fails with the same error.

> The warning in the previous email suggets you're running a fairly old
> kernel.. there is probably something handled incorrectly during the fs
> init process.  Exactly which kernel are you running?

2.6.32 - apparently the latest in Debian stable.  I figured this was workable 
since ceph.com offers packages for Debian stable.

> In any case, btrfs isn't going to work particularly well on something that
> old; I suggest running something newer (3.5 or 3.6) or switching to XFS.

Ok fair enough.  I'll have to see how practical it is to get a more recent 
kernel going, otherwise I'll go down the XFS route.

Thanks again,
Adam.