[Cluster-devel] GFS2 and uevents document

[Cluster-devel] GFS2 and uevents document

[Steven Whitehouse]

Hi,

Below is a first draft at a document explaining the uevents produced by
GFS2. I'm intending to add it under
linux-2.6/Documentation/filesystems/gfs-uevents.txt

Let me know if you spot anything thats wrong or could be better
explained,

Steve.


-------------------------------------------------------------------
                              uevents and GFS2
                             ==================

During the lifetime of a GFS2 mount, a number of uevents are generated,
this document explains what the events are and what they are used
for (by gfs_controld in gfs2-utils)

A list of GFS2 uevents
-----------------------

1. ADD

The ADD event occurs at mount time. It will always be the first
uevent generated by the newly created filesystem. If the mount
is successful, an ONLINE uevent will follow, if it is not successful
then a REMOVE uevent will follow.

The ADD uevent has two environment variables: SPECTATOR=[0|1]
and RDONLY=[0|1] which specify the spectator (no journal assigned,
implies a read-only mount) and read-only status of the filesystem
respectively.

2. ONLINE

The ONLINE uevent is generated after a successful mount or remount. It
has the same environment variables as the ADD uevent. The ONLINE
uevent, along with the two environment variables for spectator and
rdonly are a relatively recent addition (2.6.32-rc+) and will not
be generated by older kernels.

3. CHANGE

The CHANGE uevent is used in two places. One is when reporting the
sucessful mount of the filesystem by the first node (FIRSTMOUNT=Done).
This is used as a signal by gfs_controld that it is then ok for other
nodes in the cluster to mount the filesystem.

The other CHANGE uevent is used to inform of the completion
of journal recovery for one of the filesystems journals. It has
two environment variables, JID= which specifies the journal id which
has just been recovered, and RECOVERY=[Done|Failed] to indicate the
success (or otherwise) of the operation. These uevents are generated
for every journal recovered whether it is during the initial mount
process or as the result of gfs_controld requesting a specific journal
recovery via the /sys/fs/gfs2/<fsname>/lock_module/recovery file.

Because the CHANGE uevent was used (in early versions of gfs_controld)
without checking the environment variables to discover the state, we
cannot add any more functions to it without running the risk of
someone using an older version of the user tools and breaking their
cluster. For this reason the ONLINE uevent was used when adding a new
uevent for a successful mount or remount.

4. OFFLINE

The OFFLINE uevent is only generated due to filesystem errors and is used
as part of the "withdraw" mechanism. Currently this doesn't give any
information about what the error is, which is something that needs to
be fixed.

5. REMOVE

The REMOVE uevent is generated at the end of an unsuccessful mount
or at the end of a umount of the filesystem. All REMOVE uevents will
have been preceeded by at least an ADD uevent for the same fileystem,
and unlike the other uevents is generated automatically by the kernel's
kobject subsystem.


Information common to all GFS2 uevents (uevent environment variables)
----------------------------------------------------------------------

1. LOCKTABLE=

The LOCKTABLE is a string, as supplied on the mount command
line (locktable=) or via fstab. It is used as a filesystem label
as well as providing the information for a lock_dlm mount to be
able to join the cluster.

2. LOCKPROTO=

The LOCKPROTO is a string, again its value depends on that set
on the mount command line, or via fstab. It will be either
lock_nolock or lock_dlm. In the future other lock managers
may be supported.

3. JOURNALID=

If a journal is in use by the filesystem (journals are not
assigned for spectator mounts) then this will give the
numeric journal id in all GFS2 uevents.

4. UUID=

With recent versions of gfs2-utils, mkfs.gfs2 writes a UUID
into the filesystem superblock. If it exists, then this will
be included in every uevent relating to the filesystem.

[Cluster-devel] GFS2 and uevents document

[David Teigland]

On Thu, Aug 13, 2009 at 03:58:00PM +0100, Steven Whitehouse wrote:
> 2. ONLINE
> 
> The ONLINE uevent is generated after a successful mount or remount. It
> has the same environment variables as the ADD uevent. The ONLINE
> uevent, along with the two environment variables for spectator and
> rdonly are a relatively recent addition (2.6.32-rc+) and will not
> be generated by older kernels.

The standard way to know if a mount is successful is to look at the return
value from mount(2).  Is a second way really needed?  Will whatever userspace
code that needs the result need to wait for and collect both results?  Will
both results always agree?  Will the results appear in the same order?  For
repeated mount/unmount of the same fs is there a way to match the ONLINE
uevent's to the specific mount call that produced it?

Dave

[Cluster-devel] GFS2 and uevents document

[Steven Whitehouse]

Hi,

On Thu, 2009-08-13 at 11:02 -0500, David Teigland wrote:
> On Thu, Aug 13, 2009 at 03:58:00PM +0100, Steven Whitehouse wrote:
> > 2. ONLINE
> > 
> > The ONLINE uevent is generated after a successful mount or remount. It
> > has the same environment variables as the ADD uevent. The ONLINE
> > uevent, along with the two environment variables for spectator and
> > rdonly are a relatively recent addition (2.6.32-rc+) and will not
> > be generated by older kernels.
> 
> The standard way to know if a mount is successful is to look at the return
> value from mount(2).  Is a second way really needed?  Will whatever userspace
> code that needs the result need to wait for and collect both results?  Will
> both results always agree?  Will the results appear in the same order?  For
> repeated mount/unmount of the same fs is there a way to match the ONLINE
> uevent's to the specific mount call that produced it?
> 
> Dave
> 

To answer the questions... yes, it will be successful only if mount is
successful as its done right at the end of the mount, rather than the
beginning (where the ADD is sent). There is no need in general to
collect both results - just filter the events that you are interested
in. It is also done after all the remount argument checking has
occurred, so its reporting the actual state of the filesystem as opposed
to what the user might have requested. If there is an error in the
remount arguments then the remount will fail and no ONLINE uevent will
be sent. We could potentially change that so that it would report the
status of the command and send ONLINE events even if the request is
going to fail, but I'm not sure it would be useful at all.

One reason for adding the ONLINE event is that we might want to add an
"errors=remount-ro" option in the future, and it would be a way for
gfs_controld to keep track of any resulting state change.

The results will always appear in the same order with respect to each
other (on the same fs) and the mount/umount commands. You can use the
variables which are sent with every message (LOCKPROTO= and LOCKTABLE=)
to match the ADD and the ONLINE uevents. The idea is that by adding
these variables to all messages it should always be possible to match
the messages belonging to a single gfs2 mount, whatever type it is. We
might want to consider adding some identifier for the block device too
if that proves useful.

It might be possible to match the specific mount call to the uevent -
I'd need to think about that and why we need to do it. At the worst we
could add a new variable on ADD to print the pid or something if its
required. The interface is intended to be extendible by adding new
environment variables to messages as required, for example I'd like to
add some additional information to the OFFLINE event at some stage so
that we can detail more exactly the error which has occurred. So if we
do need some way to track mount calls by adding another variable, it
shouldn't be a big issue.

I'm not really expecting that many userland programs will want to make
use of the GFS2 uevents other than gfs_controld, but even so documenting
them seems to be a good plan. I can also see possible applications of
the OFFLINE uevent in triggering other events, such as notifying an
administrator that something has gone wrong, for example. But thats a
little way off yet I think,

Steve.

[Cluster-devel] GFS2 and uevents document

[Bob Peterson]

----- "Steven Whitehouse" <swhiteho@redhat.com> wrote:
| Hi,
| 
| Below is a first draft at a document explaining the uevents produced
| by
| GFS2. I'm intending to add it under
| linux-2.6/Documentation/filesystems/gfs-uevents.txt
| 
| Let me know if you spot anything thats wrong or could be better
| explained,
| 
| Steve.

Hi Steve,

Good writeup.  Here are some minor suggestions.

Bob Peterson
--
--- /home/msp/rpeterso/gfs_uevents.orig.txt	2009-08-13 10:02:25.000000000 -0500
+++ /home/msp/rpeterso/gfs_uevents.bobs.txt	2009-08-13 10:10:33.000000000 -0500
@@ -1,9 +1,9 @@
                               uevents and GFS2
                              ==================
 
-During the lifetime of a GFS2 mount, a number of uevents are generated,
-this document explains what the events are and what they are used
-for (by gfs_controld in gfs2-utils)
+During the lifetime of a GFS2 mount, a number of uevents are generated.
+This document explains what the events are and what they are used
+for (by gfs_controld in gfs2-utils).
 
 A list of GFS2 uevents
 -----------------------
@@ -12,26 +12,26 @@ A list of GFS2 uevents
 
 The ADD event occurs at mount time. It will always be the first
 uevent generated by the newly created filesystem. If the mount
-is successful, an ONLINE uevent will follow, if it is not successful
+is successful, an ONLINE uevent will follow.  If it is not successful
 then a REMOVE uevent will follow.
 
 The ADD uevent has two environment variables: SPECTATOR=[0|1]
-and RDONLY=[0|1] which specify the spectator (no journal assigned,
-implies a read-only mount) and read-only status of the filesystem
-respectively.
+and RDONLY=[0|1] that specify the spectator status (a read-only mount
+with no journal assigned), and read-only (with journal assigned) status
+of the filesystem respectively.
 
 2. ONLINE
 
 The ONLINE uevent is generated after a successful mount or remount. It
 has the same environment variables as the ADD uevent. The ONLINE
 uevent, along with the two environment variables for spectator and
-rdonly are a relatively recent addition (2.6.32-rc+) and will not
+RDONLY are a relatively recent addition (2.6.32-rc+) and will not
 be generated by older kernels.
 
 3. CHANGE
 
 The CHANGE uevent is used in two places. One is when reporting the
-sucessful mount of the filesystem by the first node (FIRSTMOUNT=Done).
+successful mount of the filesystem by the first node (FIRSTMOUNT=Done).
 This is used as a signal by gfs_controld that it is then ok for other
 nodes in the cluster to mount the filesystem.
 
@@ -40,7 +40,7 @@ of journal recovery for one of the files
 two environment variables, JID= which specifies the journal id which
 has just been recovered, and RECOVERY=[Done|Failed] to indicate the
 success (or otherwise) of the operation. These uevents are generated
-for every journal recovered whether it is during the initial mount
+for every journal recovered, whether it is during the initial mount
 process or as the result of gfs_controld requesting a specific journal
 recovery via the /sys/fs/gfs2/<fsname>/lock_module/recovery file.
 
@@ -79,7 +79,7 @@ able to join the cluster.
 
 2. LOCKPROTO=
 
-The LOCKPROTO is a string, again its value depends on that set
+The LOCKPROTO is a string, and its value depends on what is set
 on the mount command line, or via fstab. It will be either
 lock_nolock or lock_dlm. In the future other lock managers
 may be supported.
@@ -93,7 +93,7 @@ numeric journal id in all GFS2 uevents.
 4. UUID=
 
 With recent versions of gfs2-utils, mkfs.gfs2 writes a UUID
-into the filesystem superblock. If it exists, then this will
+into the filesystem superblock. If it exists, this will
 be included in every uevent relating to the filesystem.