Re: [ANNOUNCE] Minneapolis Cluster Summit, July 29-30

Daniel Phillips
2004-07-08T01:10:24Z

On Wednesday 07 July 2004 14:16, Lars Marowsky-Bree wrote:
> On 2004-07-06T17:34:51, Daniel Phillips <phillips@redhat.com> said:
> > > And the "industry" was very reluctant
> > > too. Which meant that everybody spend ages talking and not much
> > > happening.
> >
> > We're showing up with loads of Sistina code this time. It's up to
> > everybody else to ante up, and yes, I see there's more code out
> > there. It's going to be quite a summer reading project.
>
> Yeah, I wish you the best. There's always been quite a bit of code to
> show, but that alone didn't convince people ;-) I've certainly grown
> a bit more experienced / cynical during that time. (Which, according
> to Oscar Wilde, is the same anyway ;)

OK, what I've learned from the discussion so far is, we need to avoid
getting stuck too much on the HA aspects and focus more on the
cluster/performance side for now. There are just too many entrenched
positions on failover. Even though every component of the cluster is
designed to fail over, that's just a small part of what we have to deal
with:

- Cluster Volume management
- Cluster configuration management
- Cluster membership/quorum
- Node Fencing
- Parallel cluster filesystems with local semantics
- Distributed Locking
- Cluster mirror block device
- Cluster snapshot block device
- Cluster administration interface, including volume managment
- Cluster resource balancing
- bits I forgot to mention

Out of that, we need to pick the three or four items we're prepared to
address immediately, that we can obviously share between at least two
known cluster filesystems, and get them onto lkml for peer review.
Trying to push the whole thing as one lump has never worked for
anybody, and won't work in this case either. For example, the DLM is
fairly non-controversial, and important in terms of performance and
reliability. Let's start with that.

Furthermore, nobody seems interested in arguing about the cluster block
devices either, so lets just discuss how they work and get them out of
the way.

Then let's tackle the low level infrastructure, such as CCS (Cluster
Configuration System) that does a simple job, that is, it distributes
configuration files racelessly.

I heard plenty of fascinating discussion of quorum strategies last
night, and have a number of papers to read as a result. But that's a
diversion: it can and must be pluggable. We just need to agree on how
the plugs work, a considerably less ambitious task.

In general, the principle is: the less important it is, the more
argument there will be about it. Defer that, make it pluggable, call
it policy, push it to user space, and move on. We need to agree on the
basics so that we can manage network volumes with cluster filesystems
on top of them.

> > I can believe it. What I have just done with my cluster snapshot
> > target over the last couple of weeks is, removed _every_ dependency
> > on cluster infrastructure and moved the one remaining essential
> > interface to user space.
>
> Is there a KS presentation on this? I didn't get invited to KS and
> will just be allowed in for OLS, but I'll be around town already...

There will be a BOF at OLS, "Cluster Infrastructure". Since I didn't
get a KS invite either and what remains is more properly lkml stuff
anyway, I will go canoing with Matt O'Keefe during KS as planned. We
already did the necessary VFS fixups over the last year (save the
non-critical flock patch, which is now in play) so there is nothing
much left to beg Linus for. There are additional VFS hooks that would
be nice to have for optimization, but they can wait, people will
appreciate them more that way ;)

The non-vfs cluster infrastructure just uses the normal module API,
except for a couple of places in the DM cluster block devices where
I've allowed myself some creative license, easily undone. Again, this
is lkml material, not KS stuff.

> > It looks like fencing is more of an issue, because having several
> > node fencing systems running at the same time in ignorance of each
> > other is deeply wrong. We can't just wave our hands at this by
> > making it pluggable, we need to settle on one that works and use
> > it. I'll humbly suggest that Sistina is furthest along in this
> > regard.
>
> Your fencing system is fine with me; based on the assumption that you
> always have to fence a failed node, you are doing the right thing.
> However, the issues are more subtle when this is no longer true, and
> in a 1:1 how do you arbitate who is allowed to fence?

Good question. Since two-node clusters are my primary interest at the
moment, I need some answers. I think the current plan is: they try to
fence each other, winner take all. Each node will introspect to decide
if it's in good enough shape to do the job itself, then go try to fence
the other one. Alternatively, they can be configured so that one has
more votes than the other, if somebody wants that broken arrangement.

This is my dim recollection, I'll have more to say when I've actually
hooked my stuff up to it. There are others with plenty of experience
in this, see below.

> > Cluster resource management is the least advanced of the components
> > that our Red Hat Sistina group has to offer, mainly because it is
> > seen as a matter of policy, and so the pressing need at this state
> > is to provide suitable hooks.
> >
> > "STOMITH" :) Yes, exactly. Global load balancing is another big
> > item, i.e., which node gets assigned the job of running a
> > particular service, which means you need to know how much of each
> > of several different kinds of resources a particular service
> > requires, and what the current resource usage profile is for each
> > node on the cluster. Rik van Riel is taking a run at this.
>
> Right, cluster resource management is one of the things where I'm
> quite happy with the approach the new heartbeat resource manager is
> heading down (or up, I hope ;).

Combining heartbeat and resource management sounds like a good idea.
Currently, we have them separate and since I have not tried it myself
yet, I'll reserve comment. Dave Teigland would be more than happy to
wax poetic, though.

> > It's a huge, scary problem. We _must_ be able to plug in different
> > solutions, all the way from completely manual to completely
> > automagic, and we have to be able to handle more than one at once.
>
> You can plug multiple ones as long as they are managing independent
> resources, obviously. However, if the CRM is the one which ultimately
> decides whether a node needs to be fenced or not - based on its
> knowledge of which resources it owns or could own - this gets a lot
> more scary still...

We do not see the CRM as being involved in fencing at present, though I
can see why perhaps it ought to be. The resource manager that Lon
Hohberger is cooking up is scriptable and rule-driven. I'm sure we
could spend 100% of the available time on that alone. My strategy is,
I send my manually-configurable cluster bits to Lon and he hooks them
in so everything is automagic, then I look at how much the end result
sucks/doesn't suck.

There's some philosophy at work here: I feel that any cluster device
that requires elaborate infrastructure and configuration to run is
broken. If you can set the cluster devices up manually and they depend
only on existing kernel interfaces, they're more likely to get unit
testing. At the same time, these devices have to fit well into a
complex infrastructure, therefore the manual interface can be driven
equally well by a script or C program, and there is one tiny but
crucial additional hook to allow for automatic reconnection to the
cluster if something bad happens, or if the resource manager just feels
the need to reorganize things.

So while I'm rambling here, I'll mention that the resource manager (or
anybody else) can just summarily cut the block target's pipe and the
block target will politely go ask for a new one. No IOs will be
failed, nothing will break, no suspend needed, just one big breaker
switch to throw. This of course depends on the target using a pipe
(socket) to communicate with the cluster, but even if I do switch to
UDP, I'll still keep at least one pipe around, just because it makes
the target so easy to control.

It didn't start this way. The first prototype had a couple thousand
lines of glue code to work with various possible infrastructures. Now
that's all gone and there are just two pipes left, one to local user
space for cluster management and the other to somewhere out on the
cluster for synchronization. It's now down to 30% of the original size
and runs faster as a bonus. All cluster interfaces are "read/write",
except for one ioctl to reconnect a broken pipe.

> > Incidently, there is already a nice crosssection of the cluster
> > community on the way to sunny Minneapolis for the July meeting.
> > We've reached about 50% capacity, and we have quorum, I think :-)
>
> Uhm, do I have to be frightened of being fenced? ;)

Only if you drink too much of that kluster Koolaid

Regards,

Daniel