Read/Write NFS I/O performance degraded by FLUSH_STABLE page flushing

Brian R Cowan
2009-04-30T20:12:21Z

Hello all,

This is my first post, so please be gentle.... I have been working with a
customer who is attempting to build their product in ClearCase dynamic
views on Linux. When they went from Red hat Enterprise Linux 4 (update 5)
to Red Hat Enterprise Linux 5 (Update 2), their build performance degraded
dramatically. When troubleshooting the issue, we noticed that links on
RHEL 5 caused an incredible number of "STABLE" 4kb nfs writes even though
the storage we were writing to was EXPLICITLY mounted async. (This made
RHEL 5 nearly 5x slower than RHEL 4.5 in this area...)

On consultation with some internal resources, we found this change in the
2.6 kernel:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=ab0a3dbedc51037f3d2e22ef67717a987b3d15e2

In here it looks like the NFS client is forcing sync writes any time a
write of less than the NFS write size occurs. We tested this hypothesis by
setting the write size to 2KB. The "STABLE" writes went away and link
times came back down out of the stratosphere. We built a modified kernel
based on the RHEL 5.2 kernel (that ONLY backed out of this change) and we
got a 33% improvement in overall build speeds. In my case, I see almost
identical build times between the 2 OS's when we use this modified kernel
on RHEL 5.

Now, why am I posing this to the list? I need to understand *why* that
change was made. On the face of it, simply backing out that patch would be
perfect. I'm paranoid. I want to make sure that this is the ONLY reason:
"/* For single writes, FLUSH_STABLE is more efficient */ "

It seems more accurate to say that they *aren't* more efficient, but
rather are "safer, but slower."

I know that this is a 3+ year old update, but RHEL 4 is based on a 2.4
kernel, and SLES 9 is based on something in the same ballpark. And our
customers see problems when they go to SLES 10/RHEL 5 from the prior major
distro version.

=================================================================
Brian Cowan
Advisory Software Engineer
ClearCase Customer Advocacy Group (CAG)
Rational Software
IBM Software Group
81 Hartwell Ave
Lexington, MA

Phone: 1.781.372.3580
Web: http://www.ibm.com/software/rational/support/

Please be sure to update your PMR using ESR at
http://www-306.ibm.com/software/support/probsub.html or cc all
correspondence to sw_support@us.ibm.com to be sure your PMR is updated in
case I am not available.