mkfs on osd - failed in 0.47

mkfs on osd - failed in 0.47

[Sławomir Skowron]

Ubuntu precise:
Linux obs-10-177-66-4 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10
20:39:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

# mount
/dev/sdc on /vol0/data/osd.0 type ext4
(rw,noexec,nodev,noatime,nodiratime,user_xattr,data=writeback,barrier=0)

# ceph-osd -i 0 --mkjournal --mkfs --monmap /tmp/monmap
2012-05-21 22:36:54.150374 7f65fbc0b780 -1 filestore(/vol0/data/osd.0)
leveldb db created
2012-05-21 22:36:54.152699 7f65fbc0b780 -1 filestore(/vol0/data/osd.0)
Extended attributes don't appear to work. Got error (2) No such file
or directory. If you are using ext3 or ext4, be sure to mount the
underlying file system with the 'user_xattr' option.
2012-05-21 22:36:54.152729 7f65fbc0b780 -1 OSD::mkfs: couldn't mount
FileStore: error -95
2012-05-21 22:36:54.152761 7f65fbc0b780 -1  ** ERROR: error creating
empty object store in /vol0/data/osd.0: (95) Operation not supported

# ls -la /vol0/data/osd.0
total 524328
drwxr-xr-x  4 root root      4096 May 21 22:36 .
drwxr-xr-x 29 root root      4096 May 21 22:34 ..
drwxr-xr-x  3 root root      4096 May 21 22:36 current
-rw-r--r--  1 root root        37 May 21 22:36 fsid
-rw-r--r--  1 root root 536870912 May 21 22:36 journal
drwx------  2 root root     16384 May  9 15:06 lost+found
-rw-r--r--  1 root root         4 May 21 22:36 store_version

Is anyone have some idea ??

-- 
-----
Pozdrawiam

Sławek "sZiBis" Skowron
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mkfs on osd - failed in 0.47

[Stefan Priebe]

Am 21.05.2012 22:41, schrieb Sławomir Skowron:
> # ceph-osd -i 0 --mkjournal --mkfs --monmap /tmp/monmap
> 2012-05-21 22:36:54.150374 7f65fbc0b780 -1 filestore(/vol0/data/osd.0)
> leveldb db created
> 2012-05-21 22:36:54.152699 7f65fbc0b780 -1 filestore(/vol0/data/osd.0)
> Extended attributes don't appear to work. Got error (2) No such file
> or directory. If you are using ext3 or ext4, be sure to mount the
> underlying file system with the 'user_xattr' option.
> 2012-05-21 22:36:54.152729 7f65fbc0b780 -1 OSD::mkfs: couldn't mount
> FileStore: error -95
> 2012-05-21 22:36:54.152761 7f65fbc0b780 -1  ** ERROR: error creating
> empty object store in /vol0/data/osd.0: (95) Operation not supported

I get the same (v0.47) while using XFS and trying to restore an osd:

#~: ceph-osd -c /etc/ceph/ceph.conf -i 1 --mkfs --monmap /tmp/monmap
2012-05-21 22:47:34.151171 7f9090a80780 -1 filestore(/srv) leveldb db 
created
2012-05-21 22:47:34.539570 7f9090a80780 -1 filestore(/srv) Extended 
attributes don't appear to work. Got error (2) No such file or 
directory. If you are using ext3 or ext4, be sure to mount the 
underlying file system with the 'user_xattr' option.
2012-05-21 22:47:34.539618 7f9090a80780 -1 OSD::mkfs: couldn't mount 
FileStore: error -95
2012-05-21 22:47:34.539642 7f9090a80780 -1  ** ERROR: error creating 
empty object store in /srv: (95) Operation not supported

Stefan
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mkfs on osd - failed in 0.47

[Gregory Farnum]

Are you actually running as root? (ie, right perms?)
What version of Ceph are you using? If you pulled and built binaries
off of master over the weekend, I believe it was broken for a few
hours in a way that will manifest somewhat like this.

On Mon, May 21, 2012 at 1:49 PM, Stefan Priebe <s.priebe@profihost.ag> wrote:
> Am 21.05.2012 22:41, schrieb Sławomir Skowron:
>
>> # ceph-osd -i 0 --mkjournal --mkfs --monmap /tmp/monmap
>> 2012-05-21 22:36:54.150374 7f65fbc0b780 -1 filestore(/vol0/data/osd.0)
>> leveldb db created
>> 2012-05-21 22:36:54.152699 7f65fbc0b780 -1 filestore(/vol0/data/osd.0)
>> Extended attributes don't appear to work. Got error (2) No such file
>> or directory. If you are using ext3 or ext4, be sure to mount the
>> underlying file system with the 'user_xattr' option.
>> 2012-05-21 22:36:54.152729 7f65fbc0b780 -1 OSD::mkfs: couldn't mount
>> FileStore: error -95
>> 2012-05-21 22:36:54.152761 7f65fbc0b780 -1  ** ERROR: error creating
>> empty object store in /vol0/data/osd.0: (95) Operation not supported
>
>
> I get the same (v0.47) while using XFS and trying to restore an osd:
>
> #~: ceph-osd -c /etc/ceph/ceph.conf -i 1 --mkfs --monmap /tmp/monmap
> 2012-05-21 22:47:34.151171 7f9090a80780 -1 filestore(/srv) leveldb db
> created
> 2012-05-21 22:47:34.539570 7f9090a80780 -1 filestore(/srv) Extended
> attributes don't appear to work. Got error (2) No such file or directory. If
> you are using ext3 or ext4, be sure to mount the underlying file system with
> the 'user_xattr' option.
> 2012-05-21 22:47:34.539618 7f9090a80780 -1 OSD::mkfs: couldn't mount
> FileStore: error -95
> 2012-05-21 22:47:34.539642 7f9090a80780 -1  ** ERROR: error creating empty
> object store in /srv: (95) Operation not supported
>
> Stefan
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mkfs on osd - failed in 0.47

[Sławomir Skowron]

Yes on root. 30 minutes ago on 0.46 same operation works.

ii  ceph                             0.47-1precise
distributed storage and file system
ii  ceph-common                      0.47-1precise              common
utilities to mount and interact with a ceph filesystem
ii  ceph-fuse                        0.47-1precise
FUSE-based client for the Ceph distributed file system
ii  libcephfs1                       0.47-1precise              Ceph
distributed file system client library
ii  python-ceph                      0.47-1precise              Python
libraries for the Ceph distributed filesystem

Repo:

deb http://ceph.com/debian/ precise main


On Mon, May 21, 2012 at 10:54 PM, Gregory Farnum <greg@inktank.com> wrote:
> Are you actually running as root? (ie, right perms?)
> What version of Ceph are you using? If you pulled and built binaries
> off of master over the weekend, I believe it was broken for a few
> hours in a way that will manifest somewhat like this.
>
> On Mon, May 21, 2012 at 1:49 PM, Stefan Priebe <s.priebe@profihost.ag> wrote:
>> Am 21.05.2012 22:41, schrieb Sławomir Skowron:
>>
>>> # ceph-osd -i 0 --mkjournal --mkfs --monmap /tmp/monmap
>>> 2012-05-21 22:36:54.150374 7f65fbc0b780 -1 filestore(/vol0/data/osd.0)
>>> leveldb db created
>>> 2012-05-21 22:36:54.152699 7f65fbc0b780 -1 filestore(/vol0/data/osd.0)
>>> Extended attributes don't appear to work. Got error (2) No such file
>>> or directory. If you are using ext3 or ext4, be sure to mount the
>>> underlying file system with the 'user_xattr' option.
>>> 2012-05-21 22:36:54.152729 7f65fbc0b780 -1 OSD::mkfs: couldn't mount
>>> FileStore: error -95
>>> 2012-05-21 22:36:54.152761 7f65fbc0b780 -1  ** ERROR: error creating
>>> empty object store in /vol0/data/osd.0: (95) Operation not supported
>>
>>
>> I get the same (v0.47) while using XFS and trying to restore an osd:
>>
>> #~: ceph-osd -c /etc/ceph/ceph.conf -i 1 --mkfs --monmap /tmp/monmap
>> 2012-05-21 22:47:34.151171 7f9090a80780 -1 filestore(/srv) leveldb db
>> created
>> 2012-05-21 22:47:34.539570 7f9090a80780 -1 filestore(/srv) Extended
>> attributes don't appear to work. Got error (2) No such file or directory. If
>> you are using ext3 or ext4, be sure to mount the underlying file system with
>> the 'user_xattr' option.
>> 2012-05-21 22:47:34.539618 7f9090a80780 -1 OSD::mkfs: couldn't mount
>> FileStore: error -95
>> 2012-05-21 22:47:34.539642 7f9090a80780 -1  ** ERROR: error creating empty
>> object store in /srv: (95) Operation not supported
>>
>> Stefan
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
-----
Pozdrawiam

Sławek "sZiBis" Skowron
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mkfs on osd - failed in 0.47

[Stefan Priebe]

Am 21.05.2012 22:58, schrieb Sławomir Skowron:
> Yes on root. 30 minutes ago on 0.46 same operation works.
>
...
> Repo:
>
> deb http://ceph.com/debian/ precise main

Same to me except i'm using debian squeeze.

Stefan
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mkfs on osd - failed in 0.47

[Sage Weil]

On Mon, 21 May 2012, Stefan Priebe wrote:
> Am 21.05.2012 22:58, schrieb Sawomir Skowron:
> > Yes on root. 30 minutes ago on 0.46 same operation works.
> > 
> ...
> > Repo:
> > 
> > deb http://ceph.com/debian/ precise main
> 
> Same to me except i'm using debian squeeze.

I just pushed a fix for this to the stable branch.  Will build a v0.47.1 
shortly.

sage

Re: mkfs on osd - failed in 0.47

[Sławomir Skowron]

Great Thanks.

On Mon, May 21, 2012 at 11:24 PM, Sage Weil <sage@inktank.com> wrote:
> On Mon, 21 May 2012, Stefan Priebe wrote:
>> Am 21.05.2012 22:58, schrieb Sawomir Skowron:
>> > Yes on root. 30 minutes ago on 0.46 same operation works.
>> >
>> ...
>> > Repo:
>> >
>> > deb http://ceph.com/debian/ precise main
>>
>> Same to me except i'm using debian squeeze.
>
> I just pushed a fix for this to the stable branch.  Will build a v0.47.1
> shortly.
>
> sage



-- 
-----
Pozdrawiam

Sławek "sZiBis" Skowron
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mkfs on osd - failed in 0.47

[Sławomir Skowron]

One more thing:

=== osd.0 ===
2012-05-22 10:14:09.801059 7ffc4414a780 -1 filestore(/vol0/data/osd.0)
leveldb db created
2012-05-22 10:14:09.804227 7ffc4414a780 -1 filestore(/vol0/data/osd.0)
limited size xattrs -- enable filestore_xattr_use_omap
2012-05-22 10:14:09.804250 7ffc4414a780 -1 OSD::mkfs: couldn't mount
FileStore: error -95
2012-05-22 10:14:09.804364 7ffc4414a780 -1  ** ERROR: error creating
empty object store in /vol0/data/osd.0: (95) Operation not supported
failed: '/sbin/mkcephfs -d /tmp/mkcephfs.T8O2L4YPFH --init-daemon osd.0'

/dev/sdb on /vol0/data/osd.0 type ext4
(rw,noexec,nodev,noatime,nodiratime,user_xattr,data=writeback,barrier=0)

ls -la osd.0
total 524332
drwxr-xr-x  4 root root      4096 May 22 10:16 .
drwxr-xr-x 29 root root      4096 May 22 10:15 ..
drwxr-xr-x  3 root root      4096 May 22 10:16 current
-rw-r--r--  1 root root        37 May 22 10:16 fsid
-rw-r--r--  1 root root 536870912 May 22 10:16 journal
drwx------  2 root root     16384 May 22 10:15 lost+found
-rw-r--r--  1 root root         4 May 22 10:16 store_version
-rwx------  1 root root         0 May 22 10:16 xattr_test


On Mon, May 21, 2012 at 11:25 PM, Sławomir Skowron <szibis@gmail.com> wrote:
> Great Thanks.
>
> On Mon, May 21, 2012 at 11:24 PM, Sage Weil <sage@inktank.com> wrote:
>> On Mon, 21 May 2012, Stefan Priebe wrote:
>>> Am 21.05.2012 22:58, schrieb Sawomir Skowron:
>>> > Yes on root. 30 minutes ago on 0.46 same operation works.
>>> >
>>> ...
>>> > Repo:
>>> >
>>> > deb http://ceph.com/debian/ precise main
>>>
>>> Same to me except i'm using debian squeeze.
>>
>> I just pushed a fix for this to the stable branch.  Will build a v0.47.1
>> shortly.
>>
>> sage
>
>
>
> --
> -----
> Pozdrawiam
>
> Sławek "sZiBis" Skowron

-- 
-----
Pozdrawiam

Sławek Skowron
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mkfs on osd - failed in 0.47

[Greg Farnum]

On Tuesday, May 22, 2012 at 1:21 AM, Sławomir Skowron wrote:
> One more thing:
>  
> === osd.0 ===
> 2012-05-22 10:14:09.801059 7ffc4414a780 -1 filestore(/vol0/data/osd.0)
> leveldb db created
> 2012-05-22 10:14:09.804227 7ffc4414a780 -1 filestore(/vol0/data/osd.0)
> limited size xattrs -- enable filestore_xattr_use_omap
> 2012-05-22 10:14:09.804250 7ffc4414a780 -1 OSD::mkfs: couldn't mount
> FileStore: error -95
> 2012-05-22 10:14:09.804364 7ffc4414a780 -1 ** ERROR: error creating
> empty object store in /vol0/data/osd.0: (95) Operation not supported
> failed: '/sbin/mkcephfs -d /tmp/mkcephfs.T8O2L4YPFH --init-daemon osd.0'

That's because you're using something that only does small xattrs (ext4, probably?) and the system wants you to turn on "filestore_xattr_use_omap" in the config file (for OSDs). :)
I'm not sure why it's off by default — Sam?
-Greg
  
>  
> /dev/sdb on /vol0/data/osd.0 type ext4
> (rw,noexec,nodev,noatime,nodiratime,user_xattr,data=writeback,barrier=0)
>  
> ls -la osd.0
> total 524332
> drwxr-xr-x 4 root root 4096 May 22 10:16 .
> drwxr-xr-x 29 root root 4096 May 22 10:15 ..
> drwxr-xr-x 3 root root 4096 May 22 10:16 current
> -rw-r--r-- 1 root root 37 May 22 10:16 fsid
> -rw-r--r-- 1 root root 536870912 May 22 10:16 journal
> drwx------ 2 root root 16384 May 22 10:15 lost+found
> -rw-r--r-- 1 root root 4 May 22 10:16 store_version
> -rwx------ 1 root root 0 May 22 10:16 xattr_test
>  
>  
> On Mon, May 21, 2012 at 11:25 PM, Sławomir Skowron <szibis@gmail.com (mailto:szibis@gmail.com)> wrote:
> > Great Thanks.
> >  
> > On Mon, May 21, 2012 at 11:24 PM, Sage Weil <sage@inktank.com (mailto:sage@inktank.com)> wrote:
> > > On Mon, 21 May 2012, Stefan Priebe wrote:
> > > > Am 21.05.2012 22:58, schrieb Sawomir Skowron:
> > > > > Yes on root. 30 minutes ago on 0.46 same operation works.
> > > >  
> > > >  
> > > > ...
> > > > > Repo:
> > > > >  
> > > > > deb http://ceph.com/debian/ precise main
> > > >  
> > > > Same to me except i'm using debian squeeze.
> > >  
> > > I just pushed a fix for this to the stable branch. Will build a v0.47.1
> > > shortly.
> > >  
> > > sage
> >  
> >  
> >  
> > --
> > -----
> > Pozdrawiam
> >  
> > Sławek "sZiBis" Skowron
>  
> --  
> -----
> Pozdrawiam
>  
> Sławek Skowron  


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mkfs on osd - failed in 0.47

[Sławomir Skowron]

Ok, now it is clear to me.

I disable filestore_xattr_use_omap for now, and i will try to move
puppet class to xfs for a new cluster init :)

Thanks

On Tue, May 22, 2012 at 7:47 PM, Greg Farnum <greg@inktank.com> wrote:
> On Tuesday, May 22, 2012 at 1:21 AM, Sławomir Skowron wrote:
>> One more thing:
>>
>> === osd.0 ===
>> 2012-05-22 10:14:09.801059 7ffc4414a780 -1 filestore(/vol0/data/osd.0)
>> leveldb db created
>> 2012-05-22 10:14:09.804227 7ffc4414a780 -1 filestore(/vol0/data/osd.0)
>> limited size xattrs -- enable filestore_xattr_use_omap
>> 2012-05-22 10:14:09.804250 7ffc4414a780 -1 OSD::mkfs: couldn't mount
>> FileStore: error -95
>> 2012-05-22 10:14:09.804364 7ffc4414a780 -1 ** ERROR: error creating
>> empty object store in /vol0/data/osd.0: (95) Operation not supported
>> failed: '/sbin/mkcephfs -d /tmp/mkcephfs.T8O2L4YPFH --init-daemon osd.0'
>
> That's because you're using something that only does small xattrs (ext4, probably?) and the system wants you to turn on "filestore_xattr_use_omap" in the config file (for OSDs). :)
> I'm not sure why it's off by default -- Sam?
> -Greg
>
>>
>> /dev/sdb on /vol0/data/osd.0 type ext4
>> (rw,noexec,nodev,noatime,nodiratime,user_xattr,data=writeback,barrier=0)
>>
>> ls -la osd.0
>> total 524332
>> drwxr-xr-x 4 root root 4096 May 22 10:16 .
>> drwxr-xr-x 29 root root 4096 May 22 10:15 ..
>> drwxr-xr-x 3 root root 4096 May 22 10:16 current
>> -rw-r--r-- 1 root root 37 May 22 10:16 fsid
>> -rw-r--r-- 1 root root 536870912 May 22 10:16 journal
>> drwx------ 2 root root 16384 May 22 10:15 lost+found
>> -rw-r--r-- 1 root root 4 May 22 10:16 store_version
>> -rwx------ 1 root root 0 May 22 10:16 xattr_test
>>
>>
>> On Mon, May 21, 2012 at 11:25 PM, Sławomir Skowron <szibis@gmail.com (mailto:szibis@gmail.com)> wrote:
>> > Great Thanks.
>> >
>> > On Mon, May 21, 2012 at 11:24 PM, Sage Weil <sage@inktank.com (mailto:sage@inktank.com)> wrote:
>> > > On Mon, 21 May 2012, Stefan Priebe wrote:
>> > > > Am 21.05.2012 22:58, schrieb Sawomir Skowron:
>> > > > > Yes on root. 30 minutes ago on 0.46 same operation works.
>> > > >
>> > > >
>> > > > ...
>> > > > > Repo:
>> > > > >
>> > > > > deb http://ceph.com/debian/ precise main
>> > > >
>> > > > Same to me except i'm using debian squeeze.
>> > >
>> > > I just pushed a fix for this to the stable branch. Will build a v0.47.1
>> > > shortly.
>> > >
>> > > sage
>> >
>> >
>> >
>> > --
>> > -----
>> > Pozdrawiam
>> >
>> > Sławek "sZiBis" Skowron
>>
>> --
>> -----
>> Pozdrawiam
>>
>> Sławek Skowron
>
>



-- 
-----
Pozdrawiam

Sławek "sZiBis" Skowron
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html