Re: [Bugme-new] [Bug 12564] New: poor performance while preprocessing source code

[Andrew Morton <akpm@linux-foundation.org>]

(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Wed, 28 Jan 2009 11:29:52 -0800 (PST)
bugme-daemon@bugzilla.kernel.org wrote:

> http://bugzilla.kernel.org/show_bug.cgi?id=12564
> 
>            Summary: poor performance while preprocessing source code
>            Product: IO/Storage

Thanks for the report.

>            Version: 2.5
>      KernelVersion: 2.6.28.2
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: Other
>         AssignedTo: io_other@kernel-bugs.osdl.org
>         ReportedBy: steven@ngls.net
> 
> 
> Latest working kernel version: 2.6.26-rc3
> Earliest failing kernel version: 2.6.26-rc4 (or 2.6.26-rc3 + patch)

(huge performance regression in NFS)
 
> Distribution: gentoo
> 
> Hardware Environment:
>   Various 32 and 64 bit Intel and AMD machines with various
> PATA and SATA disks and various network interfaces.
> 
> Problem Description:
>   Here's my situation.  I've recently upgraded the kernels
> on ~30 computers at work (from 2.6.21 to 2.6.27).  These 
> computers are used to build and test software we develop.  
> We speed up the building process using distcc.  However,
> after the kernel upgrade, the builds are much much slower.
> The preprocessing stage seems to be at least 10 times
> slower.
>   As evidence of this slowdown I am attaching two images created
> using distccmon-gnome.  Both snapshots were taken shortly 
> after starting builds in a clean sandbox.  The only difference
> is the kernel.  "fast.png" was generated while running
> kernel 2.6.25.20.  "slow.png" was generated with 2.6.26.
> The light purple sections indicate the preprocessing times
> for each file.  
>   This slowdown is observed on both 32 and 64 bit computers
> and using either gcc or the intel compiler. (The intel compiler
> builds do not use distcc, but that are also slower.)  Strangely 
> enough, it's still faster to use an NFS mounted sandbox on a
> machine with an older kernel than the same sandbox on the local
> machine with a newer kernel.  (This suggests to me that it
> is neither a disk or network IO problem.)
>   I've used git bisect to narrow it down to a single commit:
> # bad: [b0b539739fe9b7d75002412a787cfdf4efddbc33] NFS: Ensure that 'noac'
> and/or 'actimeo=0' turn off attribute caching

And thanks for bisecting it.

> This is the first commit after v2.6.26-rc3.  I'm not an
> experienced git user, so I don't know how to narrow it down
> further.  Distccmon-gnome snapshots from the bisection
> process are similar to the ones noted above.
>   Naturally, I would like the newer kernels to have similar
> performance to the older kernels.
>   I will be attaching various items.  Let me know what other 
> information might be helpful.
> 
> Steps to reproduce:
> distccmon-gnome &
> # using a makefile setup to use distcc:
> make -j 5 all
> # note preprocessing times in distccmon
> 

Something I don't understand from this:

If your normal setup is using distcc then what role does NFS have to
play?  I mean, distcc kind-of replaces NFS in this workload.

Perhaps you could briefly describe the topology/data-flow/etc so that
we can see how NFS could be a bottleneck here?

Thanks.