Re: CTDB Cluster Samba on Cephfs

Re: CTDB Cluster Samba on Cephfs

[Sam Lang]

On Thu, Mar 28, 2013 at 6:32 AM, Kai Blin <kai-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 2013-03-28 09:16, Volker Lendecke wrote:
>> On Wed, Mar 27, 2013 at 10:43:36PM -0700, Matthieu Patou wrote:
>>> On 03/27/2013 10:41 AM, Marco Aroldi wrote:
>>>> Hi list, I'm trying to create a active/active Samba cluster on
>>>> top of Cephfs I would ask if Ceph fully supports CTDB at this
>>>> time.
>>> If I'm not wrong Ceph (even CephFS) do not support exporting a
>>> block device or mounting the same FS more than once whereas CTDB
>>> explicitly require that you have a distributed filesystem where
>>> the same filesystem is mounted across all the nodes.
>>
>> Is that true? I thought Ceph was one of the cluster filesystems
>> doing just that. What is Ceph if not a cluster file system?
>
> There's some problem with mounting the in-kernel cephfs driver on
> systems running the osd, iirc. I had to use the fuse-based driver to
> mount, which obviously is not too great, speed-wise.
> See http://ceph.com/docs/master/faq/#try-ceph for a better description
> of the issue.

Just to let folks know, we have a ceph vfs driver for samba that we
are testing out now.  We're planning to resolve a few of the bugs that
we're seeing presently with smbtorture, and send a pull request to the
samba repo.  If anyone wants to help with testing, let us know.  The
changes currently reside in the ceph branch of
http://github.com/ceph/samba.

-slang [inktank dev | http://www.inktank.com | http://www.ceph.com]

>
> Cheers,
> Kai
>
> - --
> Kai Blin
> Worldforge developer http://www.worldforge.org/
> Wine developer http://wiki.winehq.org/KaiBlin
> Samba team member http://www.samba.org/samba/team/
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
> Comment: Using GnuPG with undefined - http://www.enigmail.net/
>
> iEYEARECAAYFAlFUKl8ACgkQEKXX/bF2FpRjMwCghOWYnWX1RtxtygqcLYDES3EC
> nXkAoIe0aPX5AA+mF1d0rf74s0+i4gVM
> =aYfR
> -----END PGP SIGNATURE-----
> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: CTDB Cluster Samba on Cephfs

[Jeremy Allison]

On Wed, Apr 03, 2013 at 03:53:58PM -0500, Sam Lang wrote:
> On Thu, Mar 28, 2013 at 6:32 AM, Kai Blin <kai-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org> wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > On 2013-03-28 09:16, Volker Lendecke wrote:
> >> On Wed, Mar 27, 2013 at 10:43:36PM -0700, Matthieu Patou wrote:
> >>> On 03/27/2013 10:41 AM, Marco Aroldi wrote:
> >>>> Hi list, I'm trying to create a active/active Samba cluster on
> >>>> top of Cephfs I would ask if Ceph fully supports CTDB at this
> >>>> time.
> >>> If I'm not wrong Ceph (even CephFS) do not support exporting a
> >>> block device or mounting the same FS more than once whereas CTDB
> >>> explicitly require that you have a distributed filesystem where
> >>> the same filesystem is mounted across all the nodes.
> >>
> >> Is that true? I thought Ceph was one of the cluster filesystems
> >> doing just that. What is Ceph if not a cluster file system?
> >
> > There's some problem with mounting the in-kernel cephfs driver on
> > systems running the osd, iirc. I had to use the fuse-based driver to
> > mount, which obviously is not too great, speed-wise.
> > See http://ceph.com/docs/master/faq/#try-ceph for a better description
> > of the issue.
> 
> Just to let folks know, we have a ceph vfs driver for samba that we
> are testing out now.  We're planning to resolve a few of the bugs that
> we're seeing presently with smbtorture, and send a pull request to the
> samba repo.  If anyone wants to help with testing, let us know.  The
> changes currently reside in the ceph branch of
> http://github.com/ceph/samba.

One thing I noticed. Returning a struct ceph_dir_result *
cast to a DIR * struct won't work in 4.0.x as we expect
dirfd(DIR *) to return a valid file descriptor.

More as I look at it.

Jeremy.

Re: CTDB Cluster Samba on Cephfs

[Volker Lendecke]

On Wed, Apr 03, 2013 at 03:53:58PM -0500, Sam Lang wrote:
> Just to let folks know, we have a ceph vfs driver for samba that we
> are testing out now.  We're planning to resolve a few of the bugs that
> we're seeing presently with smbtorture, and send a pull request to the
> samba repo.  If anyone wants to help with testing, let us know.  The
> changes currently reside in the ceph branch of
> http://github.com/ceph/samba.

The comments indicate that ceph is completely userspace. It
might be interesting to look at the aio functions like
pread_send/recv and pwrite_send/recv. In particular for
large data transfers over SMB2 those might give a
performance boost.

With best regards,

Volker Lendecke

-- 
SerNet GmbH, Bahnhofsallee 1b, 37081 Göttingen
phone: +49-551-370000-0, fax: +49-551-370000-9
AG Göttingen, HRB 2816, GF: Dr. Johannes Loxen
http://www.sernet.de, mailto:kontakt-3ekOc4rQMZmzQB+pC5nmwQ@public.gmane.org

Re: [ceph-users] CTDB Cluster Samba on Cephfs

[Jeremy Allison]

On Wed, Apr 03, 2013 at 03:53:58PM -0500, Sam Lang wrote:
> On Thu, Mar 28, 2013 at 6:32 AM, Kai Blin <kai@samba.org> wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > On 2013-03-28 09:16, Volker Lendecke wrote:
> >> On Wed, Mar 27, 2013 at 10:43:36PM -0700, Matthieu Patou wrote:
> >>> On 03/27/2013 10:41 AM, Marco Aroldi wrote:
> >>>> Hi list, I'm trying to create a active/active Samba cluster on
> >>>> top of Cephfs I would ask if Ceph fully supports CTDB at this
> >>>> time.
> >>> If I'm not wrong Ceph (even CephFS) do not support exporting a
> >>> block device or mounting the same FS more than once whereas CTDB
> >>> explicitly require that you have a distributed filesystem where
> >>> the same filesystem is mounted across all the nodes.
> >>
> >> Is that true? I thought Ceph was one of the cluster filesystems
> >> doing just that. What is Ceph if not a cluster file system?
> >
> > There's some problem with mounting the in-kernel cephfs driver on
> > systems running the osd, iirc. I had to use the fuse-based driver to
> > mount, which obviously is not too great, speed-wise.
> > See http://ceph.com/docs/master/faq/#try-ceph for a better description
> > of the issue.
> 
> Just to let folks know, we have a ceph vfs driver for samba that we
> are testing out now.  We're planning to resolve a few of the bugs that
> we're seeing presently with smbtorture, and send a pull request to the
> samba repo.  If anyone wants to help with testing, let us know.  The
> changes currently reside in the ceph branch of
> http://github.com/ceph/samba.

Does the libceph have an async API ?

If we could plug it into the Samba async VFS pread_send/pwrite_send
API you'd get much better performance.

If libceph is thread-safe you could create your own modified
version of vfs_aio_pthread() that called the ceph backend
(although you'll need the ability to set credentials into
the ceph userspace calls to cope with seteuid changes).

Jeremy.

Re: [ceph-users] CTDB Cluster Samba on Cephfs

[Sam Lang]

On Fri, Apr 12, 2013 at 6:14 PM, Jeremy Allison <jra@samba.org> wrote:
> On Wed, Apr 03, 2013 at 03:53:58PM -0500, Sam Lang wrote:
>> On Thu, Mar 28, 2013 at 6:32 AM, Kai Blin <kai@samba.org> wrote:
>> > -----BEGIN PGP SIGNED MESSAGE-----
>> > Hash: SHA1
>> >
>> > On 2013-03-28 09:16, Volker Lendecke wrote:
>> >> On Wed, Mar 27, 2013 at 10:43:36PM -0700, Matthieu Patou wrote:
>> >>> On 03/27/2013 10:41 AM, Marco Aroldi wrote:
>> >>>> Hi list, I'm trying to create a active/active Samba cluster on
>> >>>> top of Cephfs I would ask if Ceph fully supports CTDB at this
>> >>>> time.
>> >>> If I'm not wrong Ceph (even CephFS) do not support exporting a
>> >>> block device or mounting the same FS more than once whereas CTDB
>> >>> explicitly require that you have a distributed filesystem where
>> >>> the same filesystem is mounted across all the nodes.
>> >>
>> >> Is that true? I thought Ceph was one of the cluster filesystems
>> >> doing just that. What is Ceph if not a cluster file system?
>> >
>> > There's some problem with mounting the in-kernel cephfs driver on
>> > systems running the osd, iirc. I had to use the fuse-based driver to
>> > mount, which obviously is not too great, speed-wise.
>> > See http://ceph.com/docs/master/faq/#try-ceph for a better description
>> > of the issue.
>>
>> Just to let folks know, we have a ceph vfs driver for samba that we
>> are testing out now.  We're planning to resolve a few of the bugs that
>> we're seeing presently with smbtorture, and send a pull request to the
>> samba repo.  If anyone wants to help with testing, let us know.  The
>> changes currently reside in the ceph branch of
>> http://github.com/ceph/samba.
>
> Does the libceph have an async API ?
>
> If we could plug it into the Samba async VFS pread_send/pwrite_send
> API you'd get much better performance.
>
> If libceph is thread-safe you could create your own modified
> version of vfs_aio_pthread() that called the ceph backend
> (although you'll need the ability to set credentials into
> the ceph userspace calls to cope with seteuid changes).

It doesn't have an async API, but the internals of libcephfs are all
asynchronous, so adding an async API shouldn't be hard.
-sam

>
> Jeremy.