Cluster Aware MD Driver

[Steven Dake <sdake@mvista.com>]

Brian,

The way to fix this problem is to enable the MD layer and MD tools to be 
cluster aware.  I hope sometime before March 03 to have this 
functionality implemented in 2.4 and mdadm 1.0.1.

While I have not spent a ton of time thinking about it, I'll share what 
I'm thinking in case someone else wants to get a jump start since I wont 
have time until Feb to start working on it.

What I am thinking of doing is making one node in a cluster the master, 
and all other nodes nonmasters.  The master is responsible for commiting 
superblock and resync changes to disk.  The other nodes are responsible 
for updating their status if any one node changes state.

During startup, a userland daemon runs which opens a TCP port.  Then the 
daemon attempts to connect to a list of other servers in a configuration 
file (/etc/clusterips).  On each connection, a master of the cluster is 
elected by an election algortihm.  Once the daemon makes connections, it 
culls its file descriptor list such that it only has one connection per 
server to each other server.  Whenever a state change occurs (raid set 
faulty, raid array failed, raid hot added, raid hot removed, raid start, 
raid stop, etc) the state change is transmitted via the server to all 
other server nodes.  When the server receives a state change message, it 
will handle appropriately by using the md ioctls to update its internal 
state.  There are more complications, such as heartbeating to detect 
dead nodes, etc.  All of this keeps the arrays in sync across the entire 
cluster.  The big changes here are to the mdadm package to use the local 
server to execute operations and to create a server which processes the 
state changes and executes the appropriate MD layer ioctls.

The question you might be asking is, how do you protect from each server 
overwritting similiar data such as the superblock or resync data.  The 
trick is to add an ioctl to the MD layer that turns writes on or off. 
 During master election above, the master will turn its writes on.  All 
non-masters will turn their writes off before any RAID arrays are 
started.  Also, resyncs have to be communicated across the cluster such 
that /proc/mdstat displays correct information so I believe an ioctl 
will have to be added to indicate this and allow resync from a specific 
spot in case the current master dies in a resync.  The changes to the MD 
layer should be fairly minimal and noninvasive and also work well for 
non-cluster configurations.  The writes will default to on to ensure 
that non-clusters work properly even with autostart.

The only downside of this approach is that RAID autostart can no longer 
be used.  The only solution to supporting that feature for clusters is 
to write all of the above userspace code into the kernel which would be 
a big pain and likely not accepted into the mainline.

Thanks
-steve

Brian Jackson wrote:

> At the moment if one node goes down, the rest of the nodes will 
> continue to run as expected. The only single point of failure from a 
> node point of view is the lock server. Dominik is working on 
> integrating the OpenDLM to get rid of that single point of failure. If 
> one disk in the shared storage dies, that will bring you down. I have 
> tried to stack pool on top of software raid, but the MD driver doesn't 
> play well with clusters so that won't work. We are trying to figure 
> out a way to fix this, but when GFS was originally designed the pool 
> driver(that does GFS's volume management) only had stripping(raid0). I 
> hope this answers your questions.
> --Brian Jackson
> P.S. I am not sure, but it sounds like you have some misconception 
> about how OpenGFS works. It does not use the disks on your nodes. It 
> uses some kind of shared storage.
> Vittorio Ballestra writes:
>
>> Escuse me if this is a stupid or repetead question, I've searched 
>> into the docs (?) and the mailinglists but I'm not able to understand 
>> if openGFS has some sort of fault-tollerance.
>> This are my doubts :
>> What happens if one host is down or one disk on one host is down ? 
>> Will the entire openGFS filesystem be down ? If one disk is broken 
>> and its content corrupted, will the whole openGFS be corrupted ?
>> If openGFS is not supporting any kind of fault-tollerance, can one 
>> use raid disks on each node ?
>> Thanks,
>>     V
>>
>> -------------------------------------------------------
>> This sf.net email is sponsored by:ThinkGeek
>> Welcome to geek heaven.
>> http://thinkgeek.com/sf
>> _______________________________________________
>> Opengfs-users mailing list
>> Opengfs-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/opengfs-users
>
>
>
>
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> Opengfs-users mailing list
> Opengfs-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/opengfs-users
>
>
>