ceph file system: extended attributes differ between ceph.ko and ceph-fuse

ceph file system: extended attributes differ between ceph.ko and ceph-fuse

[Andreas Bluemle]

Hi,

I am looking at ceph filesystem both via the kernel module
and ceph-fuse.

I am running on CentOS6.4 with
 - kernel 3.8.13 (for ceph.ko) and
 - ceph v0.61.4 userland components

I encounter an inconsistency between ceph.ko and ceph-fuse
regarding extended attributes:

- I have the ceph fs mounted at mount points
  /mnt/cephfs (using ceph.ko) and /mnt/cephfs-fuse

[root@rx37-2 fs_2]# mount | grep ceph
10.10.38.13:/ on /mnt/cephfs type ceph (name=admin,key=client.admin)
ceph-fuse on /mnt/cephfs-fuse type fuse.ceph-fuse
       (rw,nosuid,nodev,allow_other,default_permissions)

- I inspect the same file from the two mointpoints

[root@rx37-2 mnt]# getfattr -d -m - cephfs-fuse/ssd-pool-3/file1 
# file: cephfs-fuse/ssd-pool-3/file1
ceph.file.layout="stripe_unit=4194304 stripe_count=1
object_size=4194304 pool=SSD-group-2"

[root@rx37-2 mnt]# getfattr -d -m - cephfs/ssd-pool-3/file1 
# file: cephfs/ssd-pool-3/file1
ceph.file.layout="chunk_bytes=4194304\012stripe_count=1\012object_size=4194304\012"
ceph.layout="chunk_bytes=4194304\012stripe_count=1\012object_size=4194304\012"

Where getfattr returns info about the pool via ceph-fuse,
it doesn't show that info via ceph.ko.

- use ceph utilities to look at the file shows the missing pieces;
  at least, the results are consistent.

[root@rx37-2 mnt]# cephfs cephfs/ssd-pool-3/file1 show_layout
layout.data_pool:     3
layout.object_size:   4194304
layout.stripe_unit:   4194304
layout.stripe_count:  1
[root@rx37-2 mnt]# ceph osd dump | grep pool | \ 
              awk '{ print $1 " " $2 ": " $3 }'
pool 0: 'data'
pool 1: 'metadata'
pool 2: 'rbd'
pool 3: 'SSD-group-2'
pool 4: 'SSD-group-3'
pool 5: 'SAS-group-2'
pool 6: 'SAS-group-3'


Is that a real problem?


Best Regards

Andreas Bluemle


-- 
Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
ITXperts GmbH                       http://www.itxperts.de
Balanstrasse 73, Geb. 08            Phone: (+49) 89 89044917
D-81541 Muenchen (Germany)          Fax:   (+49) 89 89044910

Company details: http://www.itxperts.de/imprint.htm

Re: ceph file system: extended attributes differ between ceph.ko and ceph-fuse

[Gregory Farnum]

On Thu, Jul 18, 2013 at 3:49 AM, Andreas Bluemle
<andreas.bluemle@itxperts.de> wrote:
> Hi,
>
> I am looking at ceph filesystem both via the kernel module
> and ceph-fuse.
>
> I am running on CentOS6.4 with
>  - kernel 3.8.13 (for ceph.ko) and
>  - ceph v0.61.4 userland components
>
> I encounter an inconsistency between ceph.ko and ceph-fuse
> regarding extended attributes:
>
> - I have the ceph fs mounted at mount points
>   /mnt/cephfs (using ceph.ko) and /mnt/cephfs-fuse
>
> [root@rx37-2 fs_2]# mount | grep ceph
> 10.10.38.13:/ on /mnt/cephfs type ceph (name=admin,key=client.admin)
> ceph-fuse on /mnt/cephfs-fuse type fuse.ceph-fuse
>        (rw,nosuid,nodev,allow_other,default_permissions)
>
> - I inspect the same file from the two mointpoints
>
> [root@rx37-2 mnt]# getfattr -d -m - cephfs-fuse/ssd-pool-3/file1
> # file: cephfs-fuse/ssd-pool-3/file1
> ceph.file.layout="stripe_unit=4194304 stripe_count=1
> object_size=4194304 pool=SSD-group-2"
>
> [root@rx37-2 mnt]# getfattr -d -m - cephfs/ssd-pool-3/file1
> # file: cephfs/ssd-pool-3/file1
> ceph.file.layout="chunk_bytes=4194304\012stripe_count=1\012object_size=4194304\012"
> ceph.layout="chunk_bytes=4194304\012stripe_count=1\012object_size=4194304\012"
>
> Where getfattr returns info about the pool via ceph-fuse,
> it doesn't show that info via ceph.ko.
>
> - use ceph utilities to look at the file shows the missing pieces;
>   at least, the results are consistent.
>
> [root@rx37-2 mnt]# cephfs cephfs/ssd-pool-3/file1 show_layout
> layout.data_pool:     3
> layout.object_size:   4194304
> layout.stripe_unit:   4194304
> layout.stripe_count:  1
> [root@rx37-2 mnt]# ceph osd dump | grep pool | \
>               awk '{ print $1 " " $2 ": " $3 }'
> pool 0: 'data'
> pool 1: 'metadata'
> pool 2: 'rbd'
> pool 3: 'SSD-group-2'
> pool 4: 'SSD-group-3'
> pool 5: 'SAS-group-2'
> pool 6: 'SAS-group-3'
>
>
> Is that a real problem?

It's not great, but it's not impossible — right now these are
"virtual" xattrs and the clients are intercepting calls to them and
generating the data locally. However, when I look at the kclient
source code I'm seeing it print out the pool; which version are you
on?
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ceph file system: extended attributes differ between ceph.ko and ceph-fuse

[Andreas Bluemle]

On Tue, 23 Jul 2013 15:55:25 -0700
Gregory Farnum <greg@inktank.com> wrote:

> On Thu, Jul 18, 2013 at 3:49 AM, Andreas Bluemle
> <andreas.bluemle@itxperts.de> wrote:
> > Hi,
> >
> > I am looking at ceph filesystem both via the kernel module
> > and ceph-fuse.
> >
> > I am running on CentOS6.4 with
> >  - kernel 3.8.13 (for ceph.ko) and
> >  - ceph v0.61.4 userland components
> >
> > I encounter an inconsistency between ceph.ko and ceph-fuse
> > regarding extended attributes:
> >
> > - I have the ceph fs mounted at mount points
> >   /mnt/cephfs (using ceph.ko) and /mnt/cephfs-fuse
> >
> > [root@rx37-2 fs_2]# mount | grep ceph
> > 10.10.38.13:/ on /mnt/cephfs type ceph (name=admin,key=client.admin)
> > ceph-fuse on /mnt/cephfs-fuse type fuse.ceph-fuse
> >        (rw,nosuid,nodev,allow_other,default_permissions)
> >
> > - I inspect the same file from the two mointpoints
> >
> > [root@rx37-2 mnt]# getfattr -d -m - cephfs-fuse/ssd-pool-3/file1
> > # file: cephfs-fuse/ssd-pool-3/file1
> > ceph.file.layout="stripe_unit=4194304 stripe_count=1
> > object_size=4194304 pool=SSD-group-2"
> >
> > [root@rx37-2 mnt]# getfattr -d -m - cephfs/ssd-pool-3/file1
> > # file: cephfs/ssd-pool-3/file1
> > ceph.file.layout="chunk_bytes=4194304\012stripe_count=1\012object_size=4194304\012"
> > ceph.layout="chunk_bytes=4194304\012stripe_count=1\012object_size=4194304\012"
> >
> > Where getfattr returns info about the pool via ceph-fuse,
> > it doesn't show that info via ceph.ko.
> >
> > - use ceph utilities to look at the file shows the missing pieces;
> >   at least, the results are consistent.
> >
> > [root@rx37-2 mnt]# cephfs cephfs/ssd-pool-3/file1 show_layout
> > layout.data_pool:     3
> > layout.object_size:   4194304
> > layout.stripe_unit:   4194304
> > layout.stripe_count:  1
> > [root@rx37-2 mnt]# ceph osd dump | grep pool | \
> >               awk '{ print $1 " " $2 ": " $3 }'
> > pool 0: 'data'
> > pool 1: 'metadata'
> > pool 2: 'rbd'
> > pool 3: 'SSD-group-2'
> > pool 4: 'SSD-group-3'
> > pool 5: 'SAS-group-2'
> > pool 6: 'SAS-group-3'
> >
> >
> > Is that a real problem?
> 
> It's not great, but it's not impossible — right now these are
> "virtual" xattrs and the clients are intercepting calls to them and
> generating the data locally. However, when I look at the kclient
> source code I'm seeing it print out the pool; which version are you
> on?
> -Greg

Hi Greg,

I am running on kernel 3.8.13, with ceph kernel source as available
in that version.

modinfo of ceph.ko and libceph.ko:

[root ~]# modinfo ceph
filename:       /lib/modules/3.8.13/kernel/fs/ceph/ceph.ko
license:        GPL
description:    Ceph filesystem for Linux
author:         Patience Warnick <patience@newdream.net>
author:         Yehuda Sadeh <yehuda@hq.newdream.net>
author:         Sage Weil <sage@newdream.net>
srcversion:     684D8D2460CC33F44986B16
depends:        libceph
intree:         Y
vermagic:       3.8.13 SMP mod_unload modversions 

[root ~]# modinfo libceph
filename:       /lib/modules/3.8.13/kernel/net/ceph/libceph.ko
license:        GPL
description:    Ceph filesystem for Linux
author:         Patience Warnick <patience@newdream.net>
author:         Yehuda Sadeh <yehuda@hq.newdream.net>
author:         Sage Weil <sage@newdream.net>
srcversion:     62B9BD1907F7AB1CB6FC021
depends:        libcrc32c,dns_resolver
intree:         Y
vermagic:       3.8.13 SMP mod_unload modversions 

Regards

Andreas Bluemle


> Software Engineer #42 @ http://inktank.com | http://ceph.com
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 



-- 
Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
ITXperts GmbH                       http://www.itxperts.de
Balanstrasse 73, Geb. 08            Phone: (+49) 89 89044917
D-81541 Muenchen (Germany)          Fax:   (+49) 89 89044910

Company details: http://www.itxperts.de/imprint.htm
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html