GFS2 file locking issues

GFS2 file locking issues

[Kirill Kuvaldin]

Hi,

I'm trying to understand how the FS locking mechanism should work.

I'm running a clustered GFS2 across two nodes, each node is a Xen domU.
To check if locking works correctly I wrote the simple perl script below.
Basically the script opens a file, locks it to prevent others from
writing into it, writes 20 lines into it, then unlocks and closes.

------------------------------------------------------------------------------
#!/usr/bin/perl

# usage:
#  lock.pl filename uniq
#           filename - file name being written to
#           uniq     - an unique label to distinguish output of
different processes

use strict;
use Time::HiRes qw(sleep);
use FileHandle;
use Fcntl ':flock';

my $filename = $ARGV[0];
my $uniq = $ARGV[1];

open FH, '>>', $filename or die $!;
flock(FH,LOCK_EX) or die $!;
FH->autoflush(1);

for (1..20) {
        print FH "$uniq\n";
        sleep (0.1);
}

flock(FH,LOCK_UN);
close FH;
------------------------------------------------------------------------------

I ran that script on both nodes simultaneously with commands:

vm01# perl lock.pl /gfs2/testfile a
vm02# perl lock.pl /gfs2/testfile b

Then to my surprise "a"s and "b"s are randomly shuffled in the testfile like

...
a
a
b
a
b
b
...

whereas I supposed it should have been like

...
a
a
a
b
b
b
...


It looks like either locking is broken in my GFS2 (I use kernel
2.6.18-92.el5xen for CentOS 5.2) or my understanding of locking isn't
correct.
I also tested the script on GFS1 and OCFS2 and it worked well, i.e.
the output was correct.



Thanks,
Kirill

Re: GFS2 file locking issues

[steve]

Hi,

On Wed, Feb 11, 2009 at 01:55:08PM +0300, Kirill Kuvaldin wrote:
> Hi,
> 
> I'm trying to understand how the FS locking mechanism should work.
> 
> I'm running a clustered GFS2 across two nodes, each node is a Xen domU.
> To check if locking works correctly I wrote the simple perl script below.
> Basically the script opens a file, locks it to prevent others from
> writing into it, writes 20 lines into it, then unlocks and closes.
>i

It should be exactly the same on GFS and GFS2, after all the code for
flock is almost identical between the two. What mount options did you
use? Which lock manager are you using - I presume lock_dlm?

Steve.

Re: GFS2 file locking issues

[Kirill Kuvaldin]

On Wed, Feb 11, 2009 at 4:36 PM,  <steve@chygwyn.com> wrote:
> Hi,
>
> On Wed, Feb 11, 2009 at 01:55:08PM +0300, Kirill Kuvaldin wrote:
>> Hi,
>>
>> I'm trying to understand how the FS locking mechanism should work.
>>
>> I'm running a clustered GFS2 across two nodes, each node is a Xen domU.
>> To check if locking works correctly I wrote the simple perl script below.
>> Basically the script opens a file, locks it to prevent others from
>> writing into it, writes 20 lines into it, then unlocks and closes.
>>i
>
> It should be exactly the same on GFS and GFS2, after all the code for
> flock is almost identical between the two. What mount options did you
> use?

No specific options used, just mount -t gfs2 /dev/gfsc/lvol0 /gfs2...

> Which lock manager are you using - I presume lock_dlm?
>

Yes, the FS was created using to be used with lock_dlm. I'm not sure
if it is actually used for GFS2. How can I check that?

Kirill

Re: GFS2 file locking issues

[steve]

Hi,

On Wed, Feb 11, 2009 at 05:37:11PM +0300, Kirill Kuvaldin wrote:
> On Wed, Feb 11, 2009 at 4:36 PM,  <steve@chygwyn.com> wrote:
> > Hi,
> >
> > On Wed, Feb 11, 2009 at 01:55:08PM +0300, Kirill Kuvaldin wrote:
> >> Hi,
> >>
> >> I'm trying to understand how the FS locking mechanism should work.
> >>
> >> I'm running a clustered GFS2 across two nodes, each node is a Xen domU.
> >> To check if locking works correctly I wrote the simple perl script below.
> >> Basically the script opens a file, locks it to prevent others from
> >> writing into it, writes 20 lines into it, then unlocks and closes.
> >>i
> >
> > It should be exactly the same on GFS and GFS2, after all the code for
> > flock is almost identical between the two. What mount options did you
> > use?
> 
> No specific options used, just mount -t gfs2 /dev/gfsc/lvol0 /gfs2...
> 
> > Which lock manager are you using - I presume lock_dlm?
> >
> 
> Yes, the FS was created using to be used with lock_dlm. I'm not sure
> if it is actually used for GFS2. How can I check that?
> 
> Kirill
cat /proc/mounts

It should be listed in the options. That will also tell you if localflocks
has been set as well. We do know of a bug in the GFS/GFS2 flock code though,
it ought to be using an interruptible wait and it doesn't at the moment.
Otherwise I don't know of any other issues.

Which kernel version are you using?

Steve.

Re: GFS2 file locking issues

[Kirill Kuvaldin]

On Wed, Feb 11, 2009 at 8:18 PM,  <steve@chygwyn.com> wrote:
> cat /proc/mounts

/dev/mapper/gfsc-lvol0 /gfs2 gfs2 rw,hostdata=jid=0:id=720897:first=1 0 0

I also tried mounting explicitly specifying lockproto=lock_dlm, but it
didn't help.

Is my understanding of locking correct after all? Is that the case if
the process makes flock on a file, no other processes including ones
running on other cluster nodes could obtain a lock until the first
writer releases it?

>
> It should be listed in the options. That will also tell you if localflocks
> has been set as well. We do know of a bug in the GFS/GFS2 flock code though,
> it ought to be using an interruptible wait and it doesn't at the moment.
> Otherwise I don't know of any other issues.
>
> Which kernel version are you using?

2.6.18-92.el5xen (from CentOS 5.2)


Kirill

Re: GFS2 file locking issues

[steve]

Hi,

On Wed, Feb 11, 2009 at 08:57:12PM +0300, Kirill Kuvaldin wrote:
> On Wed, Feb 11, 2009 at 8:18 PM,  <steve@chygwyn.com> wrote:
> > cat /proc/mounts
> 
> /dev/mapper/gfsc-lvol0 /gfs2 gfs2 rw,hostdata=jid=0:id=720897:first=1 0 0
> 
> I also tried mounting explicitly specifying lockproto=lock_dlm, but it
> didn't help.
> 
> Is my understanding of locking correct after all? Is that the case if
> the process makes flock on a file, no other processes including ones
> running on other cluster nodes could obtain a lock until the first
> writer releases it?
>
Yes, thats how it is supposed to work.
 
> >
> > It should be listed in the options. That will also tell you if localflocks
> > has been set as well. We do know of a bug in the GFS/GFS2 flock code though,
> > it ought to be using an interruptible wait and it doesn't at the moment.
> > Otherwise I don't know of any other issues.
> >
> > Which kernel version are you using?
> 
> 2.6.18-92.el5xen (from CentOS 5.2)
> 
> 
> Kirill

The CentOS 5.2 kernel is rather old[*], I'd suggest using a
more recent kernel and gfs2-utils. Either something derrived
from a recent upstream (Linus) kernel such as Fedora, or
5.3 and upwards for CentOS. It might not cure this specific
issue, but it will cure a lot of other issues which you might
run across,

Steve.

[*] It was not an officially supported feature even in Red Hat
Enterprise Linux of the same version number. As a result any
bugs found have been fixed in 5.3 and up, rather than in 5.2
update releases.