[Ocfs2-devel] Clustered Samba/CTDB on ocfs2 ?

[Ocfs2-devel] Clustered Samba/CTDB on ocfs2 ?

[tridge]

Is anyone interested in trying clustered Samba on ocfs2?

A group of us have been working on a new design for clustered Samba
for the last 6 months or so, and we now have something that is
suitable for wider testing. One of the interesting aspects of the
design is that it should work on top of just about any clustered
filesystem, and it doesn't rely on any specific DLM or other
infrastructure.

So I'd be very interested to hear if anyone tries it with ocfs2. So
far its only been tested with GPFS (and it performed very well).

See http://wiki.samba.org/index.php/CTDB_Setup for setup details.

Cheers, Tridge

[Ocfs2-devel] Clustered Samba/CTDB on ocfs2 ?

[Mark Fasheh]

On Thu, May 03, 2007 at 06:30:21AM +1000, tridge@samba.org wrote:
> Is anyone interested in trying clustered Samba on ocfs2?

Definitely - working well with Samba is one of the things that Ocfs2 is
interested in.


> A group of us have been working on a new design for clustered Samba
> for the last 6 months or so, and we now have something that is
> suitable for wider testing. One of the interesting aspects of the
> design is that it should work on top of just about any clustered
> filesystem, and it doesn't rely on any specific DLM or other
> infrastructure.

Does it require the file system to support cluster-aware fcntl locking?
That's the one thing that Ocfs2 doesn't do yet.
	--Mark

--
Mark Fasheh
Senior Software Developer, Oracle
mark.fasheh@oracle.com

[Ocfs2-devel] Clustered Samba/CTDB on ocfs2 ?

[tridge]

Mark,

 > Definitely - working well with Samba is one of the things that Ocfs2 is
 > interested in.

great!

 > Does it require the file system to support cluster-aware fcntl locking?
 > That's the one thing that Ocfs2 doesn't do yet.

no, it doesn't need fcntl locking in the filesystem.

Basically CTDB is a clustered version of the tdb database in Samba. It
has its own DLM, and its own communications system (can be either TCP
or infiniband).

Not all of the Samba internal databases can use CTDB yet, so for those
that aren't yet using CTDB they should be stored on a filesystem that
supports fcntl locking. Eventually all the Samba databases will be
able to use CTDB, at which time Samba should be making no fcntl lock
requests to the cluster filesystem at all. In the meantime, you can
still test Samba on ocfs2, but if you (for example) change passwords
on two nodes at the same time then there is a slight risk of data
corruption in the passsword database. That's because the tdbsam
database is not yet using CTDB. You could avoid that particular
problem by using a ldap server for your password database.

I assume ocfs2 is data coherent? That is the main requirement from the
cluster filesystem. If you are not data coherent then clustered Samba
will still work, but a user who writes a file to one node may get old
data if they fetch it on another node.

Cheers, Tridge

[Ocfs2-devel] Clustered Samba/CTDB on ocfs2 ?

[Mark Fasheh]

On Thu, May 03, 2007 at 07:12:09AM +1000, tridge@samba.org wrote:
> no, it doesn't need fcntl locking in the filesystem.
> 
> Basically CTDB is a clustered version of the tdb database in Samba. It
> has its own DLM, and its own communications system (can be either TCP
> or infiniband).

Ok, excellent. This should be a good fit then :)


> Not all of the Samba internal databases can use CTDB yet, so for those
> that aren't yet using CTDB they should be stored on a filesystem that
> supports fcntl locking. Eventually all the Samba databases will be
> able to use CTDB, at which time Samba should be making no fcntl lock
> requests to the cluster filesystem at all. In the meantime, you can
> still test Samba on ocfs2, but if you (for example) change passwords
> on two nodes at the same time then there is a slight risk of data
> corruption in the passsword database. That's because the tdbsam
> database is not yet using CTDB. You could avoid that particular
> problem by using a ldap server for your password database.

Ok - understood. Out of curiousity, where is this stuff in its lifetime
(i.e., alpha, aeta, post release, etc). Once we verify that things are
working with Ocfs2 as expected, we could start pointing people asking about
Samba to the CTDB stuff.


> I assume ocfs2 is data coherent? That is the main requirement from the
> cluster filesystem. If you are not data coherent then clustered Samba
> will still work, but a user who writes a file to one node may get old
> data if they fetch it on another node.

Yeah - we handle data coherency between nodes. O_DIRECT gets you around some
of those guarantees, so mixing it with buffered I/O is generally discouraged.
	--Mark

--
Mark Fasheh
Senior Software Developer, Oracle
mark.fasheh@oracle.com

[Ocfs2-devel] Clustered Samba/CTDB on ocfs2 ?

[tridge]

Mark,

 > Ok - understood. Out of curiousity, where is this stuff in its lifetime
 > (i.e., alpha, aeta, post release, etc). Once we verify that things are
 > working with Ocfs2 as expected, we could start pointing people asking about
 > Samba to the CTDB stuff.

It ran for the first time on a real cluster (running GPFS) at SambaXP
in Germany last week. It is progressing very rapidly though, and we
hope there will be real deployments soon.

So I guess you'd call it alpha. It doesn't yet do auto-recovery after
a node crash. We're working on that now. We want it to the point that
you can pull the power on a node in the cluster and the other nodes
continue with no data loss (that takes some fancy footwork, as we
deliberately do not replicate all the meta data between nodes in the
cluster, so we need a smart recovery system).

 > Yeah - we handle data coherency between nodes. O_DIRECT gets you around some
 > of those guarantees, so mixing it with buffered I/O is generally discouraged.
 > 	--Mark

we never use O_DIRECT, so that sounds good.

Cheers, Tridge

[Ocfs2-devel] Clustered Samba/CTDB on ocfs2 ?

[tridge]

Mark,

 > Ok - understood. Out of curiousity, where is this stuff in its lifetime
 > (i.e., alpha, aeta, post release, etc). Once we verify that things are
 > working with Ocfs2 as expected, we could start pointing people asking about
 > Samba to the CTDB stuff.

The CTDB stuff has come a long way in the last few weeks. If you or
anyone else on this list wants to do some testing with OCFS2 then I
think it would be a good time to start.

See http://ctdb.samba.org/ and
http://wiki.samba.org/index.php/CTDB_Setup#Managing_Network_Interfaces 
for some documentation other information.

Cheers, Tridge

[Ocfs2-devel] Clustered Samba/CTDB on ocfs2 ?

[Robert Wipfel]

>>> On Mon, Jun 4, 2007 at  1:02 AM, in message
<18019.47337.745002.756250@samba.org>, <tridge@samba.org> wrote: 
> Mark,
> 
>  > Ok -  understood. Out of curiousity, where is this stuff in its lifetime
>  > (i.e., alpha, aeta, post release, etc). Once we verify that things are
>  > working with Ocfs2 as expected, we could start pointing people asking 
> about
>  > Samba to the CTDB stuff.
> 
> The CTDB stuff has come a long way in the last few weeks. If you or
> anyone else on this list wants to do some testing with OCFS2 then I
> think it would be a good time to start.
> 
> See http://ctdb.samba.org/ and
> http://wiki.samba.org/index.php/CTDB_Setup#Managing_Network_Interfaces 
> for some documentation other information.

Would Samba work with something like the
cluster iptables load sharing stuff? 
http://lists.community.tummy.com/pipermail/linux-ha/2007-May/025390.html
(guess the point is to create a single system
image file server cluster)

- Robert

[Ocfs2-devel] Clustered Samba/CTDB on ocfs2 ?

[tridge]

Robert,

 > Would Samba work with something like the
 > cluster iptables load sharing stuff? 
 > http://lists.community.tummy.com/pipermail/linux-ha/2007-May/025390.html
 > (guess the point is to create a single system
 > image file server cluster)

Samba+ctdb has IP failover and load sharing builtin. It also has smart
stuff to handle startup/monitoring of any system service, so it will
do things like:

 - if the link status of the public network interface goes down, then
   the public IP of that node gets moved to a different node within a
   few seconds, and moved back when the link status is OK again

 - if an admin screws up the Samba smb.conf and ctdb is managing
   Samba, then the IP is moved. Same with if the admin stops
   portmapper and NFS etc etc.

 - ctdb does all the gratuitous arp stuff, plus does TCP 'tickle' acks
   on failover, ensuring absolutely minimum failover time without
   relying on (slow!) tcp keepalives. We run tests with windows boxes
   copying files, then pull the plug on the node that windows is
   connected to. The xcopy hardly pauses at all as it continues with
   another cluster node.

Have a look at http://samba.org/~tridge/ctdb/config/ to see the
standard config stuff in ctdb. I'm afraid the ctdb wiki is down right
now, but when it comes back there is a lot more info there.

For load sharing, we use round-robin DNS, and round-robin WINS. All
cluster nodes behave the same, so you have one name for all the
nodes. If you have the money, a load balancing switch is also a good
choice.

We eventually want to do dynamic load balancing, where Samba migrates
live connections between nodes according to demand. At the moment we
only do load balancing at connect time, which is OK, but not ideal.

Cheers, Tridge

[Ocfs2-devel] Clustered Samba/CTDB on ocfs2 ?

[Robert Wipfel]

>>> On Wed, Jun 6, 2007 at  4:33 AM, in message
<18022.36219.942944.408630@samba.org>, <tridge@samba.org> wrote: 
> Robert,
> 
>  > Would Samba work with something like the
>  > cluster iptables load sharing stuff? 
>  > http://lists.community.tummy.com/pipermail/linux- ha/2007- May/025390.html
>  > (guess the point is to create a single system
>  > image file server cluster)
> 
> Samba+ctdb has IP failover and load sharing builtin. It also has smart
> stuff to handle startup/monitoring of any system service, so it will
> do things like:

Thanks Tridge, sounds really interesting, just fwiw, and as a reminder,
it's possible to build OCFS2 clusters using iSCSI and FireWire storage,
incase anyone was concerned about the hardware requirements of building
a (Samba) cluster... ;-)

Couple of other things worth mentioning perhaps, regarding managability
of open source clusters and storage solutions:

Free commercial quality product doc for Heartbeat2:
http://lists.community.tummy.com/pipermail/linux-ha/2007-June/025566.html

Open source Linux Storage Resource Management software:
http://dev.eclipse.org/mhonarc/lists/aperi-dev/msg00485.html

>  -  if the link status of the public network interface goes down, then
>    the public IP of that node gets moved to a different node within a
>    few seconds, and moved back when the link status is OK again

Interested in how this works, I need to find time to read your Wiki... (e.g. 
binding IP addresses this could be supported by OCFS2 dlmfs, if not using
Heartbeat2...:-)

>  -  ctdb does all the gratuitous arp stuff, plus does TCP 'tickle' acks

Nice :-)

>    on failover, ensuring absolutely minimum failover time without
>    relying on (slow!) tcp keepalives. We run tests with windows boxes
>    copying files, then pull the plug on the node that windows is
>    connected to. The xcopy hardly pauses at all as it continues with
>    another cluster node.
> 
> Have a look at http://samba.org/~tridge/ctdb/config/ to see the
> standard config stuff in ctdb. I'm afraid the ctdb wiki is down right
> now, but when it comes back there is a lot more info there.
> 
> For load sharing, we use round- robin DNS, and round- robin WINS. All
> cluster nodes behave the same, so you have one name for all the
> nodes. If you have the money, a load balancing switch is also a good
> choice.
> 
> We eventually want to do dynamic load balancing, where Samba migrates
> live connections between nodes according to demand. At the moment we
> only do load balancing at connect time, which is OK, but not ideal.

Sounds really interesting/fun, thanks for the info, 

Robert

[Ocfs2-devel] Clustered Samba/CTDB on ocfs2 ?

[Mark Fasheh]

On Mon, Jun 04, 2007 at 05:02:01PM +1000, tridge@samba.org wrote:
>  > Ok - understood. Out of curiousity, where is this stuff in its lifetime
>  > (i.e., alpha, aeta, post release, etc). Once we verify that things are
>  > working with Ocfs2 as expected, we could start pointing people asking about
>  > Samba to the CTDB stuff.
> 
> The CTDB stuff has come a long way in the last few weeks. If you or
> anyone else on this list wants to do some testing with OCFS2 then I
> think it would be a good time to start.
> 
> See http://ctdb.samba.org/ and
> http://wiki.samba.org/index.php/CTDB_Setup#Managing_Network_Interfaces 
> for some documentation other information.

Hmm, that all looks a lot more involved than I had first imagined. It might
be a while before I or anyone on my team has the cycles to get a proper test
going :/

But ignoring manpower issues for a second.. What sort of tests do you think
would help to "validate" ocfs2 + clustered samba as a usable setup? I'm
thinking some sort of coherency verification against two clients mounted on
two seperate nodes at the least. Is there a particular test suite you all
run on this stuff?
	--Mark

--
Mark Fasheh
Senior Software Developer, Oracle
mark.fasheh@oracle.com

[Ocfs2-devel] Clustered Samba/CTDB on ocfs2 ?

[tridge]

Mark,

 > But ignoring manpower issues for a second.. What sort of tests do you think
 > would help to "validate" ocfs2 + clustered samba as a usable setup? I'm
 > thinking some sort of coherency verification against two clients mounted on
 > two seperate nodes at the least. Is there a particular test suite you all
 > run on this stuff?

yes, we use the smbtorture test suite, plus windows clients. For a
start I'd suggest the following tests:

  BENCH-NBENCH
  RAW-BENCH-LOCK
  RAW-BENCH-OPEN

they are part of smbtorture in Samba4. 

I'd be happy to help with testing if someone can give me access to a
OCFS2 system.

Cheers, Tridge