[Cluster-devel] Re: Why the gfs2 performance regressed?

[Cluster-devel] Re: Why the gfs2 performance regressed?

[Steven Whitehouse]

Hi,

On Wed, 2008-01-02 at 15:53 +0800, Cheng Renquan wrote:
> hello, Steven:
> 
> I have tested gfs2 in the out-of-box RHEL51 and in the latest gfs2-nmw
> git repository, finding that the latest gfs2 performance regressed.
> 
> the testing environment is RHEL5.1, default kernel + gfs2 + samba, I
> use the preinstalled samba-3.0.25b to share a gfs2 mouted direcotry,
> on Windows clients I've tested its performance is very good! it could
> be up to 100MB/s on writing, almost up to the limit of the local
> Gigabyte ether network,
> 
> Now I want to test the latest gfs2 from gfs2-nmw.git repository, I
> just compiled the gfs2-nmw kernel to replace the default, using the
> .config from /boot/config-$(uname -r)
> 
> But I found the performance is bad:
> 
> Writers on Windows can only up to 6 MB/s, at the time I noticed on the linux:
> 
> 1. from this I know the smbd thread 3627 is serving the Windows client
> 192.168.76.226:
> # lsof -nPi4
> smbd      3627 fstest   22u  IPv4   7806       TCP
> 192.168.76.200:445->192.168.76.226:1369 (ESTABLISHED)
> 
> 2. list all opening files of thread 3627, finding that /mnt/gfs2/x.dat
> is currently accessed by Windows client testing utilities through
> samba:
> # lsof -nP -a -p 3627
> smbd    3627 fstest   18uR  REG      253,0 328704000    66392 /mnt/gfs2/x.dat
> smbd    3627 fstest   22u  IPv4       7806                TCP
> 192.168.76.200:445->192.168.76.226:1369 (ESTABLISHED)
> 
> 3. strace it and record every system call time, output the results to a file:
> # strace -T -p 3627 -o fcntl64.3267
> 
> 4. sometime later interrupt it, and analysis the result.
> 
> 5. from the result, I found some fcntl64 system call consumes 0.9 second:
> fcntl64(18, F_GETLK64, {type=F_UNLCK, whence=SEEK_SET,
> start=325263360, len=65536, pid=0}) = 0 <0.915883>
> fcntl64(18, F_GETLK64, {type=F_UNLCK, whence=SEEK_SET, start=22093824,
> len=65536, pid=0}) = 0 <0.916389>
> ...
> 
> 6. if straced thread 3627 without output redirected, the fcntl64 will
> appear as an apparent pause.
> 
> 7. Since the default-kernel+gfs2+samba serves high efficiently, and
> the latest gfs2-nmw does not, this should be some problem in gfs2-nmw?
> 
> 8. I noted sometimes the thread 3627 will becomes D state
> (uninterruptible), I use /proc/sysrq-trigger to record the call trace
> in the kernel space:
> smbd          D 00000000  2080  3627   3620
>        00000000 00200082 00000001 00000000 c042148e 00000000 19b1e094 000002b6
>        f68f92c0 f68f94f0 c3616d80 00000000 f6dda040 000004ab 00000000 00000003
>        f605ae9c f8c54340 c043bc87 f34fa0c0 f605af14 f605ae9c f488fe00 f8c4fbea
> Call Trace:
>  [<c042148e>] __wake_up_common+0x32/0x5c
>  [<c043bc87>] prepare_to_wait+0x24/0x3f
>  [<f8c4fbea>] gdlm_plock_get+0xeb/0x16b [lock_dlm]
>  [<c043bb3d>] autoremove_wake_function+0x0/0x35
>  [<f8c4faff>] gdlm_plock_get+0x0/0x16b [lock_dlm]
>  [<f8b7b5d5>] gfs2_lm_plock_get+0x30/0x39 [gfs2]
>  [<f8b81003>] gfs2_lock+0x78/0xb5 [gfs2]
>  [<f8b80f8b>] gfs2_lock+0x0/0xb5 [gfs2]
>  [<c0480a51>] vfs_test_lock+0x18/0x23
>  [<c0480d95>] fcntl_getlk64+0x5f/0x108
>  [<c047e54b>] sys_fcntl64+0x38/0x6d
>  [<c0404dde>] sysenter_past_esp+0x5f/0x85
> 
> 9. Someone could tell me if there is a problem of gfs2-nmw or not? Or
> the configuration of mine has problems?
> 
> the two attachments are the results of strace and dmesg, fcntl64.3267
> and dmesg.1
> 
Are you running single node? if so then use lock_nolock rather than
lock_dlm as it will be much faster for fcntl locks. Even if you intend
to run as a cluster eventually, a single node comparison against
lock_nolock would be useful to try and eliminate some possibilities,

Steve.

[Cluster-devel] Re: Why the gfs2 performance regressed?

[Cheng Renquan]

On Jan 2, 2008 6:19 PM, Steven Whitehouse <swhiteho@redhat.com> wrote:
> Are you running single node? if so then use lock_nolock rather than
> lock_dlm as it will be much faster for fcntl locks. Even if you intend
> to run as a cluster eventually, a single node comparison against
> lock_nolock would be useful to try and eliminate some possibilities,
No, I'm using a two nodes environment, the /mnt/gfs2 are gfs2 mounting
points on both two nodes, and they are both samba shared folders.
And in fact, the two nodes are using a clustered LVM from the same
iSCSI device, so dlm would be the only lock manager?

Actually What I cannot explain is merely that the only difference with
the kernel: from 2.6.18-53.el5 (stocked with RHEL51) to latest
gfs2-nmw.git, I don't know why.

-- 
Denis Cheng

[Cluster-devel] Re: Why the gfs2 performance regressed?

[Steven Whitehouse]

Hi,

On Thu, 2008-01-03 at 00:44 +0800, Cheng Renquan wrote:
> On Jan 2, 2008 6:19 PM, Steven Whitehouse <swhiteho@redhat.com> wrote:
> > Are you running single node? if so then use lock_nolock rather than
> > lock_dlm as it will be much faster for fcntl locks. Even if you intend
> > to run as a cluster eventually, a single node comparison against
> > lock_nolock would be useful to try and eliminate some possibilities,
> No, I'm using a two nodes environment, the /mnt/gfs2 are gfs2 mounting
> points on both two nodes, and they are both samba shared folders.
> And in fact, the two nodes are using a clustered LVM from the same
> iSCSI device, so dlm would be the only lock manager?
> 
> Actually What I cannot explain is merely that the only difference with
> the kernel: from 2.6.18-53.el5 (stocked with RHEL51) to latest
> gfs2-nmw.git, I don't know why.
> 

There was a performance regression with the -nmw tree relating to a set
of patches which I removed from the tree this morning. That would make
it slower, but not by the amount that you are seeing, and also it would
affect only "normal" I/O and not fcntl locks.

Is there a point in the history of the  -nmw git tree which you know is
ok? Perhaps it would be possible to bisect the tree to find the problem
patch?

Steve.

[Cluster-devel] Re: Why the gfs2 performance regressed?

[Denis Cheng]

On Jan 3, 2008 12:52 AM, Steven Whitehouse <swhiteho@redhat.com> wrote:
> There was a performance regression with the -nmw tree relating to a set
> of patches which I removed from the tree this morning. That would make
> it slower, but not by the amount that you are seeing, and also it would
> affect only "normal" I/O and not fcntl locks.
>
> Is there a point in the history of the  -nmw git tree which you know is
> ok? Perhaps it would be possible to bisect the tree to find the problem
> patch?
I've pulled from gfs2-2.6-nmw.git on this morning, (Thu Jan  3
03:12:29 UTC 2008), and what you said should be synced here,
but it didn't solve the problem, even more serious, some warning
kernel OOPS messages will appear on umount.gfs2,

I compiled an older kernel release and 2.6.21.7 has been tested to be
good, samba writing tests report 70 MB/s again.

I'm now testing a newer kernel release.

>
> Steve.

-- 
Denis Cheng

[Cluster-devel] Re: Why the gfs2 performance regressed?

[Denis Cheng]

On Jan 3, 2008 3:27 PM, Denis Cheng <crquan@gmail.com> wrote:
> I compiled an older kernel release and 2.6.21.7 has been tested to be
> good, samba writing tests report 70 MB/s again.
>
> I'm now testing a newer kernel release.
After some bisecting work, now 2.6.22.15 is verified to be able to
support samba high performance, while 2.6.23 cannot,

-- 
Denis Cheng

[Cluster-devel] Re: Why the gfs2 performance regressed?

[rae l]

On Jan 3, 2008 5:36 PM, Denis Cheng <crquan@gmail.com> wrote:
> After some bisecting work, now 2.6.22.15 is verified to be able to
> support samba high performance, while 2.6.23 cannot,
git bisecting work not finished, now v2.6.22 good and v2.6.23-rc1 bad.

2.6.22-g3bd858ab also good.

-- 
Denis Cheng

[Cluster-devel] Re: Why the gfs2 performance regressed?

[Steven Whitehouse]

Hi,

On Mon, 2008-01-07 at 17:48 +0800, rae l wrote:
> On Jan 3, 2008 5:36 PM, Denis Cheng <crquan@gmail.com> wrote:
> > After some bisecting work, now 2.6.22.15 is verified to be able to
> > support samba high performance, while 2.6.23 cannot,
> git bisecting work not finished, now v2.6.22 good and v2.6.23-rc1 bad.
> 
> 2.6.22-g3bd858ab also good.
> 

One of the patches I see in this range is:

commit 60446067ba7a8e890a91db3b4a7436fe0ebd2dee
Author: Marc Eshel <eshel@almaden.ibm.com>
Date:   Mon Jan 15 18:33:36 2007 -0500

    gfs2: stop giving out non-cluster-coherent leases


Now I wonder if samba is using fcntl locks rather than (incorrect, in
the clustered GFS2 case, since it doesn't support them) leases and that
is now why you are seeing the slow down. You could try (for testing
purposes only - don't do this with any important data) setting the
localflocks flag on mount and see if that makes a difference to the
speed.

If it does make a difference, then the problem is just that GFS2 doesn't
support leases in a clustered environment. If it makes no difference,
then it points more towards there being a slow-down in the I/O side,
which seems odd since my general impression is that I/O has been getting
faster recently,

Steve.

[Cluster-devel] Re: Why the gfs2 performance regressed?

[rae l]

On Jan 7, 2008 7:48 PM, Steven Whitehouse <swhiteho@redhat.com> wrote:
> commit 60446067ba7a8e890a91db3b4a7436fe0ebd2dee
> Author: Marc Eshel <eshel@almaden.ibm.com>
> Date:   Mon Jan 15 18:33:36 2007 -0500
>
>     gfs2: stop giving out non-cluster-coherent leases
>
>
> Now I wonder if samba is using fcntl locks rather than (incorrect, in
> the clustered GFS2 case, since it doesn't support them) leases and that
> is now why you are seeing the slow down. You could try (for testing
> purposes only - don't do this with any important data) setting the
> localflocks flag on mount and see if that makes a difference to the
> speed.
>
> If it does make a difference, then the problem is just that GFS2 doesn't
> support leases in a clustered environment. If it makes no difference,
> then it points more towards there being a slow-down in the I/O side,
> which seems odd since my general impression is that I/O has been getting
> faster recently,
that does make a difference indeed. and the git bisect work also finished,

60446067ba7a8e890a91db3b4a7436fe0ebd2dee is first bad commit

then I tested the HEAD of gfs2-2.6-nmw:
with 'localflocks' mount option, everything goes well and samba serves
with high performance;
without that mount option, something goes bad (umount.gfs2 would cause
kernel panic) and samba serves slow down.

then I reverted that commit in the HEAD, everything goes well again,
umount.gfs2 run and quilt normally.

So now it's time to determine: is 'localflocks' mount option
nessessary for all clustered deployment from then on? or we can
determine 60446067 is a bad commit really?

>
> Steve.

-- 
Denis Cheng

[Cluster-devel] Re: Why the gfs2 performance regressed?

[Steven Whitehouse]

Hi,

On Tue, 2008-01-08 at 15:38 +0800, rae l wrote:
> On Jan 7, 2008 7:48 PM, Steven Whitehouse <swhiteho@redhat.com> wrote:
> > commit 60446067ba7a8e890a91db3b4a7436fe0ebd2dee
> > Author: Marc Eshel <eshel@almaden.ibm.com>
> > Date:   Mon Jan 15 18:33:36 2007 -0500
> >
> >     gfs2: stop giving out non-cluster-coherent leases
> >
> >
> > Now I wonder if samba is using fcntl locks rather than (incorrect, in
> > the clustered GFS2 case, since it doesn't support them) leases and that
> > is now why you are seeing the slow down. You could try (for testing
> > purposes only - don't do this with any important data) setting the
> > localflocks flag on mount and see if that makes a difference to the
> > speed.
> >
> > If it does make a difference, then the problem is just that GFS2 doesn't
> > support leases in a clustered environment. If it makes no difference,
> > then it points more towards there being a slow-down in the I/O side,
> > which seems odd since my general impression is that I/O has been getting
> > faster recently,
> that does make a difference indeed. and the git bisect work also finished,
> 
> 60446067ba7a8e890a91db3b4a7436fe0ebd2dee is first bad commit
> 
> then I tested the HEAD of gfs2-2.6-nmw:
> with 'localflocks' mount option, everything goes well and samba serves
> with high performance;
> without that mount option, something goes bad (umount.gfs2 would cause
> kernel panic) and samba serves slow down.
> 
> then I reverted that commit in the HEAD, everything goes well again,
> umount.gfs2 run and quilt normally.
> 
> So now it's time to determine: is 'localflocks' mount option
> nessessary for all clustered deployment from then on? or we can
> determine 60446067 is a bad commit really?
> 
> >
> > Steve.
> 
The patch is correct. The problem seems to be that your test was
previously incorrect since it was using local leases, whereas it
shouldn't have been using leases at all since they were not supported
across the cluster. The solution appears to be that either we need to
improve the performance of the fcntl locks, or, and I suspect better
still in the samba case, we need to look into how we might implement
cluster leases,

Steve.

[Cluster-devel] Re: Why the gfs2 performance regressed?

[rae l]

On Jan 8, 2008 5:01 PM, Steven Whitehouse <swhiteho@redhat.com> wrote:
> Hi,
>
>
> On Tue, 2008-01-08 at 15:38 +0800, rae l wrote:
> > On Jan 7, 2008 7:48 PM, Steven Whitehouse <swhiteho@redhat.com> wrote:
> > > commit 60446067ba7a8e890a91db3b4a7436fe0ebd2dee
> > > Author: Marc Eshel <eshel@almaden.ibm.com>
> > > Date:   Mon Jan 15 18:33:36 2007 -0500
> > >
> > >     gfs2: stop giving out non-cluster-coherent leases
> > >
> > >
> > > Now I wonder if samba is using fcntl locks rather than (incorrect, in
> > > the clustered GFS2 case, since it doesn't support them) leases and that
> > > is now why you are seeing the slow down. You could try (for testing
> > > purposes only - don't do this with any important data) setting the
> > > localflocks flag on mount and see if that makes a difference to the
> > > speed.
> > >
> > > If it does make a difference, then the problem is just that GFS2 doesn't
> > > support leases in a clustered environment. If it makes no difference,
> > > then it points more towards there being a slow-down in the I/O side,
> > > which seems odd since my general impression is that I/O has been getting
> > > faster recently,
> > that does make a difference indeed. and the git bisect work also finished,
> >
> > 60446067ba7a8e890a91db3b4a7436fe0ebd2dee is first bad commit
> >
> > then I tested the HEAD of gfs2-2.6-nmw:
> > with 'localflocks' mount option, everything goes well and samba serves
> > with high performance;
> > without that mount option, something goes bad (umount.gfs2 would cause
> > kernel panic) and samba serves slow down.
> >
> > then I reverted that commit in the HEAD, everything goes well again,
> > umount.gfs2 run and quilt normally.
> >
> > So now it's time to determine: is 'localflocks' mount option
> > nessessary for all clustered deployment from then on? or we can
> > determine 60446067 is a bad commit really?
> >
> > >
> > > Steve.
> >
> The patch is correct. The problem seems to be that your test was
> previously incorrect since it was using local leases, whereas it
> shouldn't have been using leases at all since they were not supported
> across the cluster. The solution appears to be that either we need to
> improve the performance of the fcntl locks, or, and I suspect better
> still in the samba case, we need to look into how we might implement
> cluster leases,
But when there was no 'setlease' at all in gfs2 before this commit,
(RHEL51 also no setlease), it could also work well on clusters with
multiple server.
So now it confused me that what's the defect of gfs2 without 'setlease'?
On the other hand, what's the purpose of 'setlease' method on a filesystem?

Could someone give some pointers on design of setlease?

-- 
Denis Cheng