Re: [Bugme-new] [Bug 12564] New: poor performance while preprocessing source code

Re: [Bugme-new] [Bug 12564] New: poor performance while preprocessing source code

[Andrew Morton]

(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Wed, 28 Jan 2009 11:29:52 -0800 (PST)
bugme-daemon@bugzilla.kernel.org wrote:

> http://bugzilla.kernel.org/show_bug.cgi?id=12564
> 
>            Summary: poor performance while preprocessing source code
>            Product: IO/Storage

Thanks for the report.

>            Version: 2.5
>      KernelVersion: 2.6.28.2
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: Other
>         AssignedTo: io_other@kernel-bugs.osdl.org
>         ReportedBy: steven@ngls.net
> 
> 
> Latest working kernel version: 2.6.26-rc3
> Earliest failing kernel version: 2.6.26-rc4 (or 2.6.26-rc3 + patch)

(huge performance regression in NFS)
 
> Distribution: gentoo
> 
> Hardware Environment:
>   Various 32 and 64 bit Intel and AMD machines with various
> PATA and SATA disks and various network interfaces.
> 
> Problem Description:
>   Here's my situation.  I've recently upgraded the kernels
> on ~30 computers at work (from 2.6.21 to 2.6.27).  These 
> computers are used to build and test software we develop.  
> We speed up the building process using distcc.  However,
> after the kernel upgrade, the builds are much much slower.
> The preprocessing stage seems to be at least 10 times
> slower.
>   As evidence of this slowdown I am attaching two images created
> using distccmon-gnome.  Both snapshots were taken shortly 
> after starting builds in a clean sandbox.  The only difference
> is the kernel.  "fast.png" was generated while running
> kernel 2.6.25.20.  "slow.png" was generated with 2.6.26.
> The light purple sections indicate the preprocessing times
> for each file.  
>   This slowdown is observed on both 32 and 64 bit computers
> and using either gcc or the intel compiler. (The intel compiler
> builds do not use distcc, but that are also slower.)  Strangely 
> enough, it's still faster to use an NFS mounted sandbox on a
> machine with an older kernel than the same sandbox on the local
> machine with a newer kernel.  (This suggests to me that it
> is neither a disk or network IO problem.)
>   I've used git bisect to narrow it down to a single commit:
> # bad: [b0b539739fe9b7d75002412a787cfdf4efddbc33] NFS: Ensure that 'noac'
> and/or 'actimeo=0' turn off attribute caching

And thanks for bisecting it.

> This is the first commit after v2.6.26-rc3.  I'm not an
> experienced git user, so I don't know how to narrow it down
> further.  Distccmon-gnome snapshots from the bisection
> process are similar to the ones noted above.
>   Naturally, I would like the newer kernels to have similar
> performance to the older kernels.
>   I will be attaching various items.  Let me know what other 
> information might be helpful.
> 
> Steps to reproduce:
> distccmon-gnome &
> # using a makefile setup to use distcc:
> make -j 5 all
> # note preprocessing times in distccmon
> 

Something I don't understand from this:

If your normal setup is using distcc then what role does NFS have to
play?  I mean, distcc kind-of replaces NFS in this workload.

Perhaps you could briefly describe the topology/data-flow/etc so that
we can see how NFS could be a bottleneck here?

Thanks.

Re: [Bugme-new] [Bug 12564] New: poor performance while preprocessing source code

[Jeff Layton]

On Wed, 28 Jan 2009 13:29:42 -0800
Andrew Morton <akpm@linux-foundation.org> wrote:

> 
> (switched to email.  Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
> 
> On Wed, 28 Jan 2009 11:29:52 -0800 (PST)
> bugme-daemon@bugzilla.kernel.org wrote:
> 
> > http://bugzilla.kernel.org/show_bug.cgi?id=12564
> > 
> >            Summary: poor performance while preprocessing source code
> >            Product: IO/Storage
> 
> Thanks for the report.
> 
> >            Version: 2.5
> >      KernelVersion: 2.6.28.2
> >           Platform: All
> >         OS/Version: Linux
> >               Tree: Mainline
> >             Status: NEW
> >           Severity: normal
> >           Priority: P1
> >          Component: Other
> >         AssignedTo: io_other@kernel-bugs.osdl.org
> >         ReportedBy: steven@ngls.net
> > 
> > 
> > Latest working kernel version: 2.6.26-rc3
> > Earliest failing kernel version: 2.6.26-rc4 (or 2.6.26-rc3 + patch)
> 
> (huge performance regression in NFS)
>  
> > Distribution: gentoo
> > 
> > Hardware Environment:
> >   Various 32 and 64 bit Intel and AMD machines with various
> > PATA and SATA disks and various network interfaces.
> > 
> > Problem Description:
> >   Here's my situation.  I've recently upgraded the kernels
> > on ~30 computers at work (from 2.6.21 to 2.6.27).  These 
> > computers are used to build and test software we develop.  
> > We speed up the building process using distcc.  However,
> > after the kernel upgrade, the builds are much much slower.
> > The preprocessing stage seems to be at least 10 times
> > slower.
> >   As evidence of this slowdown I am attaching two images created
> > using distccmon-gnome.  Both snapshots were taken shortly 
> > after starting builds in a clean sandbox.  The only difference
> > is the kernel.  "fast.png" was generated while running
> > kernel 2.6.25.20.  "slow.png" was generated with 2.6.26.
> > The light purple sections indicate the preprocessing times
> > for each file.  
> >   This slowdown is observed on both 32 and 64 bit computers
> > and using either gcc or the intel compiler. (The intel compiler
> > builds do not use distcc, but that are also slower.)  Strangely 
> > enough, it's still faster to use an NFS mounted sandbox on a
> > machine with an older kernel than the same sandbox on the local
> > machine with a newer kernel.  (This suggests to me that it
> > is neither a disk or network IO problem.)
> >   I've used git bisect to narrow it down to a single commit:
> > # bad: [b0b539739fe9b7d75002412a787cfdf4efddbc33] NFS: Ensure that 'noac'
> > and/or 'actimeo=0' turn off attribute caching
> 
> And thanks for bisecting it.
> 

It's a pretty small patch. The obvious question is: "What mount options
are you using on your NFS mounts?"


> > This is the first commit after v2.6.26-rc3.  I'm not an
> > experienced git user, so I don't know how to narrow it down
> > further.  Distccmon-gnome snapshots from the bisection
> > process are similar to the ones noted above.
> >   Naturally, I would like the newer kernels to have similar
> > performance to the older kernels.
> >   I will be attaching various items.  Let me know what other 
> > information might be helpful.
> > 
> > Steps to reproduce:
> > distccmon-gnome &
> > # using a makefile setup to use distcc:
> > make -j 5 all
> > # note preprocessing times in distccmon
> > 
> 
> Something I don't understand from this:
> 
> If your normal setup is using distcc then what role does NFS have to
> play?  I mean, distcc kind-of replaces NFS in this workload.
> 
> Perhaps you could briefly describe the topology/data-flow/etc so that
> we can see how NFS could be a bottleneck here?
> 
> Thanks.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


-- 
Jeff Layton <jlayton@redhat.com>

Re: [Bugme-new] [Bug 12564] New: poor performance while preprocessing source code

[Steven Patrick]

On Wed, 2009-01-28 at 17:05 -0500, Jeff Layton wrote:
> 
> It's a pretty small patch. The obvious question is: "What mount
> options
> are you using on your NFS mounts?"
> 

  Well, I didn't think that I was using any NFS mounts.
But, it turns out I was in a way.  Apparently, amd does its top
level mounts with noac.  So, where this (using an automounted path):
    cd /h/spo/work
    make -j 5
would be slow.  This (using a non-automounted path):
    cd /home/spo/work
    make -j 5
showed the faster performance I was expecting.
  I will mark my bug as invalid and decide what, if anything,
I want to do about amd.
  Thanks for pointing me in the right direction.

Steven

Re: [Bugme-new] [Bug 12564] New: poor performance while preprocessing source code

[Andrew Morton]

On Wed, 28 Jan 2009 17:02:36 -0800
Steven Patrick <steven@ngls.net> wrote:

> On Wed, 2009-01-28 at 17:05 -0500, Jeff Layton wrote:
> > 
> > It's a pretty small patch. The obvious question is: "What mount
> > options
> > are you using on your NFS mounts?"
> > 
> 
>   Well, I didn't think that I was using any NFS mounts.
> But, it turns out I was in a way.  Apparently, amd does its top
> level mounts with noac.  So, where this (using an automounted path):
>     cd /h/spo/work
>     make -j 5
> would be slow.  This (using a non-automounted path):
>     cd /home/spo/work
>     make -j 5
> showed the faster performance I was expecting.
>   I will mark my bug as invalid and decide what, if anything,
> I want to do about amd.
>   Thanks for pointing me in the right direction.
> 

This had me all mystified until I clicked on the bugzilla link:


:  ------- Comment  #9 From Trond Myklebust  2009-01-28 13:58:50  [reply] -------
: 
: If you are worried about performance, why are you using noac/actimeo=0?
: 
: The commit you point to fixed a bug in which the noac/actimeo=0 was
: using cached metadata in situations where it should have been
: retrieving the data from the server.  We fully expect a significant
: performance drop when the client is forced to send more GETATTR
: requests to the server, and as I said, that was precisely the point of
: this fix.

Fair enough.



However.  We surprised one user (yourself), and we will presumably
surprise others.  Or, worse, we will slow down people's stuff and they
won't even notice.

Should we perhaps warn people about this?  A mount-time printk telling
them that moac/actime=0 is slow?

Re: [Bugme-new] [Bug 12564] New: poor performance while preprocessing source code

[Trond Myklebust]

On Wed, 2009-01-28 at 17:37 -0800, Andrew Morton wrote:
> On Wed, 28 Jan 2009 17:02:36 -0800
> Steven Patrick <steven@ngls.net> wrote:
> 
> > On Wed, 2009-01-28 at 17:05 -0500, Jeff Layton wrote:
> > > 
> > > It's a pretty small patch. The obvious question is: "What mount
> > > options
> > > are you using on your NFS mounts?"
> > > 
> > 
> >   Well, I didn't think that I was using any NFS mounts.
> > But, it turns out I was in a way.  Apparently, amd does its top
> > level mounts with noac.  So, where this (using an automounted path):
> >     cd /h/spo/work
> >     make -j 5
> > would be slow.  This (using a non-automounted path):
> >     cd /home/spo/work
> >     make -j 5
> > showed the faster performance I was expecting.
> >   I will mark my bug as invalid and decide what, if anything,
> > I want to do about amd.
> >   Thanks for pointing me in the right direction.
> > 
> 
> This had me all mystified until I clicked on the bugzilla link:
> 
> 
> :  ------- Comment  #9 From Trond Myklebust  2009-01-28 13:58:50  [reply] -------
> : 
> : If you are worried about performance, why are you using noac/actimeo=0?
> : 
> : The commit you point to fixed a bug in which the noac/actimeo=0 was
> : using cached metadata in situations where it should have been
> : retrieving the data from the server.  We fully expect a significant
> : performance drop when the client is forced to send more GETATTR
> : requests to the server, and as I said, that was precisely the point of
> : this fix.
> 
> Fair enough.
> 
> 
> 
> However.  We surprised one user (yourself), and we will presumably
> surprise others.  Or, worse, we will slow down people's stuff and they
> won't even notice.
> 
> Should we perhaps warn people about this?  A mount-time printk telling
> them that moac/actime=0 is slow?

No. The current nfs manpages (dated 2 November 2007) have a whole
section on data and metadata coherence and about the trade-offs
involved. That section is explicitly referenced in the description of
the 'noac' mount option. It explains that 'noac' is basically about
disabling caching in order to achieve approximate cache coherent
behaviour when multiple NFS clients need to access files that are being
modified frequently on the server.

Note that 'noac' has _never_ been intended as a default NFS mount
option. The kernel default is rather to cache attributes as aggressively
as possible for scalability and performance reasons.
All other NFS mount programs therefore require 'noac' to be explicitly
set by the user. I see no reason to punish those users with extra printk
spam.

Re: [Bugme-new] [Bug 12564] New: poor performance while preprocessing source code

[Steven Patrick]

[-- Attachment #1: Type: text/plain, Size: 4759 bytes --]

On Wed, 2009-01-28 at 13:29 -0800, Andrew Morton wrote:
> (switched to email.  Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
> 
> On Wed, 28 Jan 2009 11:29:52 -0800 (PST)
> bugme-daemon@bugzilla.kernel.org wrote:
> 
> > http://bugzilla.kernel.org/show_bug.cgi?id=12564
> > 
> >            Summary: poor performance while preprocessing source code
> >            Product: IO/Storage
> 
> Thanks for the report.
> 
> >            Version: 2.5
> >      KernelVersion: 2.6.28.2
> >           Platform: All
> >         OS/Version: Linux
> >               Tree: Mainline
> >             Status: NEW
> >           Severity: normal
> >           Priority: P1
> >          Component: Other
> >         AssignedTo: io_other@kernel-bugs.osdl.org
> >         ReportedBy: steven@ngls.net
> > 
> > 
> > Latest working kernel version: 2.6.26-rc3
> > Earliest failing kernel version: 2.6.26-rc4 (or 2.6.26-rc3 + patch)
> 
> (huge performance regression in NFS)
>  
> > Distribution: gentoo
> > 
> > Hardware Environment:
> >   Various 32 and 64 bit Intel and AMD machines with various
> > PATA and SATA disks and various network interfaces.
> > 
> > Problem Description:
> >   Here's my situation.  I've recently upgraded the kernels
> > on ~30 computers at work (from 2.6.21 to 2.6.27).  These 
> > computers are used to build and test software we develop.  
> > We speed up the building process using distcc.  However,
> > after the kernel upgrade, the builds are much much slower.
> > The preprocessing stage seems to be at least 10 times
> > slower.
> >   As evidence of this slowdown I am attaching two images created
> > using distccmon-gnome.  Both snapshots were taken shortly 
> > after starting builds in a clean sandbox.  The only difference
> > is the kernel.  "fast.png" was generated while running
> > kernel 2.6.25.20.  "slow.png" was generated with 2.6.26.
> > The light purple sections indicate the preprocessing times
> > for each file.  
> >   This slowdown is observed on both 32 and 64 bit computers
> > and using either gcc or the intel compiler. (The intel compiler
> > builds do not use distcc, but that are also slower.)  Strangely 
> > enough, it's still faster to use an NFS mounted sandbox on a
> > machine with an older kernel than the same sandbox on the local
> > machine with a newer kernel.  (This suggests to me that it
> > is neither a disk or network IO problem.)
> >   I've used git bisect to narrow it down to a single commit:
> > # bad: [b0b539739fe9b7d75002412a787cfdf4efddbc33] NFS: Ensure that 'noac'
> > and/or 'actimeo=0' turn off attribute caching
> 
> And thanks for bisecting it.

  I think it might be possible to bisect further, but I
don't know how get git to do it.  

> > This is the first commit after v2.6.26-rc3.  I'm not an
> > experienced git user, so I don't know how to narrow it down
> > further.  Distccmon-gnome snapshots from the bisection
> > process are similar to the ones noted above.
> >   Naturally, I would like the newer kernels to have similar
> > performance to the older kernels.
> >   I will be attaching various items.  Let me know what other 
> > information might be helpful.
> > 
> > Steps to reproduce:
> > distccmon-gnome &
> > # using a makefile setup to use distcc:
> > make -j 5 all
> > # note preprocessing times in distccmon
> > 
> 
> Something I don't understand from this:
> 
> If your normal setup is using distcc then what role does NFS have to
> play?  I mean, distcc kind-of replaces NFS in this workload.

  I don't believe that this actually has anything to do
with NFS.  The only way I could see NFS coming into this is the
fact that I use amd to automount home directories.
  The commit mentioned above modifies just over 100 files, very
few of which appear related to NFS.  I've attached the diifs.
It appears to be a number of fixes rolled up into one patch.
This is why I believe it might be possible to continue bisecting
with git.  But, to do that I suspect I would need to use a
different repository and/or tag or version names.
  BTW, the diffs were generated with "git diff v2.6.26-rc3"
after having done the bisection.  This is why it looks to me
like this one commit includes multiple updates.  Since all
bisection iterations were bad, the diffs against the good
version must be the diffs of the one commit.  Of course, I 
reserve the right to have made a mistake or not understand 
things completely.
  There was one other detail that seemed strange.  In the
last four bisection iterations, the kernel identified
itself as -rc2 instead of -rc3.  I assumed an update
accidentally changed it back.

> 
> Perhaps you could briefly describe the topology/data-flow/etc so that
> we can see how NFS could be a bottleneck here?
> 
> Thanks.

[-- Attachment #2: diffs.txt.gz --]
[-- Type: application/x-gzip, Size: 34743 bytes --]

Re: [Bugme-new] [Bug 12564] New: poor performance while preprocessing source code

[Andrew Morton]

On Wed, 28 Jan 2009 14:43:37 -0800
Steven Patrick <steven@ngls.net> wrote:

> > > Problem Description:
> > >   Here's my situation.  I've recently upgraded the kernels
> > > on ~30 computers at work (from 2.6.21 to 2.6.27).  These 
> > > computers are used to build and test software we develop.  
> > > We speed up the building process using distcc.  However,
> > > after the kernel upgrade, the builds are much much slower.
> > > The preprocessing stage seems to be at least 10 times
> > > slower.
> > >   As evidence of this slowdown I am attaching two images created
> > > using distccmon-gnome.  Both snapshots were taken shortly 
> > > after starting builds in a clean sandbox.  The only difference
> > > is the kernel.  "fast.png" was generated while running
> > > kernel 2.6.25.20.  "slow.png" was generated with 2.6.26.
> > > The light purple sections indicate the preprocessing times
> > > for each file.  
> > >   This slowdown is observed on both 32 and 64 bit computers
> > > and using either gcc or the intel compiler. (The intel compiler
> > > builds do not use distcc, but that are also slower.)  Strangely 
> > > enough, it's still faster to use an NFS mounted sandbox on a
> > > machine with an older kernel than the same sandbox on the local
> > > machine with a newer kernel.  (This suggests to me that it
> > > is neither a disk or network IO problem.)
> > >   I've used git bisect to narrow it down to a single commit:
> > > # bad: [b0b539739fe9b7d75002412a787cfdf4efddbc33] NFS: Ensure that 'noac'
> > > and/or 'actimeo=0' turn off attribute caching
> > 
> > And thanks for bisecting it.
> 
>   I think it might be possible to bisect further, but I
> don't know how get git to do it.  

OK..

> > > This is the first commit after v2.6.26-rc3.  I'm not an
> > > experienced git user, so I don't know how to narrow it down
> > > further.  Distccmon-gnome snapshots from the bisection
> > > process are similar to the ones noted above.
> > >   Naturally, I would like the newer kernels to have similar
> > > performance to the older kernels.
> > >   I will be attaching various items.  Let me know what other 
> > > information might be helpful.
> > > 
> > > Steps to reproduce:
> > > distccmon-gnome &
> > > # using a makefile setup to use distcc:
> > > make -j 5 all
> > > # note preprocessing times in distccmon
> > > 
> > 
> > Something I don't understand from this:
> > 
> > If your normal setup is using distcc then what role does NFS have to
> > play?  I mean, distcc kind-of replaces NFS in this workload.
> 
>   I don't believe that this actually has anything to do
> with NFS.  The only way I could see NFS coming into this is the
> fact that I use amd to automount home directories.
>   The commit mentioned above modifies just over 100 files, very
> few of which appear related to NFS.  I've attached the diifs.
> It appears to be a number of fixes rolled up into one patch.
> This is why I believe it might be possible to continue bisecting
> with git.  But, to do that I suspect I would need to use a
> different repository and/or tag or version names.
>   BTW, the diffs were generated with "git diff v2.6.26-rc3"
> after having done the bisection.  This is why it looks to me
> like this one commit includes multiple updates.  Since all
> bisection iterations were bad, the diffs against the good
> version must be the diffs of the one commit.  Of course, I 
> reserve the right to have made a mistake or not understand 
> things completely.
>   There was one other detail that seemed strange.  In the
> last four bisection iterations, the kernel identified
> itself as -rc2 instead of -rc3.  I assumed an update
> accidentally changed it back.

Heaven knows.

OK, "This is the first commit after v2.6.26-rc3", so NFS is innocent. 
Presumably your git bisect landed on a big merge commit.

I'm afaid I'm not the person who can tell you how to fix this - it
hasn't happened to me in a while, but I don't know what I did to
prevent it happening :(

Here's the diffstat from your diffs.txt.gz:

 arch/m68k/configs/multi_defconfig              | 1269 -------------------------
 b/MAINTAINERS                                  |    4 
 b/Makefile                                     |    2 
 b/arch/arm/common/locomo.c                     |   66 -
 b/arch/arm/kernel/armksyms.c                   |    2 
 b/arch/arm/kernel/arthur.c                     |    2 
 b/arch/arm/mach-omap1/board-palmte.c           |    2 
 b/arch/arm/mach-omap1/board-palmz71.c          |    2 
 b/arch/arm/mach-omap2/board-2430sdp.c          |    1 
 b/arch/arm/mach-omap2/board-apollon.c          |    1 
 b/arch/arm/mach-omap2/board-generic.c          |    1 
 b/arch/arm/mach-omap2/board-h4.c               |    1 
 b/arch/arm/mach-omap2/clock.c                  |    4 
 b/arch/arm/mach-omap2/clock34xx.h              |   21 
 b/arch/arm/mach-omap2/cm-regbits-34xx.h        |    1 
 b/arch/arm/mach-omap2/mailbox.c                |   25 
 b/arch/arm/mach-omap2/prm.h                    |    2 
 b/arch/arm/mach-orion5x/dns323-setup.c         |    2 
 b/arch/arm/mach-orion5x/kurobox_pro-setup.c    |    2 
 b/arch/arm/mach-pxa/colibri.c                  |    3 
 b/arch/arm/mach-pxa/spitz.c                    |    1 
 b/arch/arm/mm/proc-arm925.S                    |    2 
 b/arch/arm/mm/proc-arm926.S                    |    2 
 b/arch/arm/mm/proc-arm940.S                    |    2 
 b/arch/arm/mm/proc-arm946.S                    |    2 
 b/arch/arm/plat-omap/clock.c                   |   10 
 b/arch/arm/plat-omap/dma.c                     |    2 
 b/arch/arm/plat-omap/mailbox.c                 |    1 
 b/arch/blackfin/mach-bf527/boards/ezkit.c      |    2 
 b/arch/m68k/Kconfig                            |    9 
 b/arch/m68k/configs/amiga_defconfig            |  159 +--
 b/arch/m68k/configs/apollo_defconfig           |  140 --
 b/arch/m68k/configs/atari_defconfig            |  143 +-
 b/arch/m68k/configs/bvme6000_defconfig         |  138 --
 b/arch/m68k/configs/hp300_defconfig            |  142 +-
 b/arch/m68k/configs/mac_defconfig              |  144 +-
 b/arch/m68k/configs/mvme147_defconfig          |  138 --
 b/arch/m68k/configs/mvme16x_defconfig          |  138 --
 b/arch/m68k/configs/q40_defconfig              |  159 +--
 b/arch/m68k/configs/sun3_defconfig             |  140 --
 b/arch/m68k/configs/sun3x_defconfig            |  140 --
 b/arch/m68k/kernel/head.S                      |    2 
 b/arch/m68k/kernel/setup.c                     |   15 
 b/arch/powerpc/platforms/pasemi/misc.c         |    7 
 b/arch/sparc64/defconfig                       |   40 
 b/arch/sparc64/mm/init.c                       |    2 
 b/arch/x86/kernel/process.c                    |   36 
 b/arch/x86/mm/pat.c                            |    2 
 b/drivers/block/amiflop.c                      |    6 
 b/drivers/block/z2ram.c                        |    2 
 b/drivers/char/snsc_event.c                    |    2 
 b/drivers/char/vme_scc.c                       |    4 
 b/drivers/i2c/busses/i2c-amd756.c              |    2 
 b/drivers/i2c/busses/i2c-nforce2.c             |   28 
 b/drivers/i2c/chips/max6875.c                  |    3 
 b/drivers/i2c/i2c-core.c                       |   22 
 b/drivers/ide/legacy/macide.c                  |    3 
 b/drivers/input/keyboard/hilkbd.c              |    4 
 b/drivers/input/misc/hp_sdc_rtc.c              |    5 
 b/drivers/input/serio/hp_sdc_mlc.c             |    5 
 b/drivers/input/serio/q40kbd.c                 |    2 
 b/drivers/media/video/cs5345.c                 |    7 
 b/drivers/media/video/cs53l32a.c               |   10 
 b/drivers/media/video/cx18/cx18-i2c.c          |    9 
 b/drivers/media/video/cx25840/cx25840-core.c   |    7 
 b/drivers/media/video/et61x251/et61x251_core.c |    2 
 b/drivers/media/video/ivtv/ivtv-i2c.c          |   13 
 b/drivers/media/video/m52790.c                 |    9 
 b/drivers/media/video/msp3400-driver.c         |   17 
 b/drivers/media/video/saa7115.c                |   40 
 b/drivers/media/video/saa7127.c                |    9 
 b/drivers/media/video/saa717x.c                |    9 
 b/drivers/media/video/sn9c102/sn9c102_core.c   |    2 
 b/drivers/media/video/tuner-core.c             |   17 
 b/drivers/media/video/upd64031a.c              |    6 
 b/drivers/media/video/upd64083.c               |    6 
 b/drivers/media/video/vp27smpx.c               |    9 
 b/drivers/media/video/wm8739.c                 |    7 
 b/drivers/media/video/wm8775.c                 |    7 
 b/drivers/media/video/zc0301/zc0301_core.c     |    2 
 b/drivers/media/video/zoran_device.c           |    2 
 b/drivers/media/video/zoran_driver.c           |    2 
 b/drivers/net/82596.c                          |    7 
 b/drivers/net/apne.c                           |    3 
 b/drivers/net/mac89x0.c                        |    3 
 b/drivers/net/macmace.c                        |    3 
 b/drivers/net/sun3lance.c                      |    3 
 b/drivers/net/wireless/atmel.c                 |    2 
 b/drivers/video/Kconfig                        |    4 
 b/drivers/video/amifb.c                        |    4 
 b/drivers/video/dnfb.c                         |    3 
 b/drivers/video/hpfb.c                         |    2 
 b/drivers/video/pxafb.c                        |    5 
 b/fs/befs/endian.h                             |    2 
 b/fs/nfs/inode.c                               |    7 
 b/include/asm-arm/arch-omap/common.h           |    4 
 b/include/asm-arm/arch-omap/control.h          |    2 
 b/include/asm-arm/arch-omap/mmc.h              |   24 
 b/include/asm-arm/arch-sa1100/irqs.h           |    2 
 b/include/asm-arm/hardware/locomo.h            |   19 
 b/include/asm-m68k/bug.h                       |    4 
 b/include/asm-m68k/io.h                        |   44 
 b/include/asm-m68k/setup.h                     |    2 
 b/include/asm-m68k/uaccess.h                   |    6 
 b/include/linux/i2c.h                          |    7 
 b/include/linux/i2c/pcf857x.h                  |    3 
 106 files changed, 833 insertions(+), 2743 deletions(-)

Now, distcc has a long track record of tripping up bugs and regressions
in the networking code - it's very clever that way.  But it doesn't
look like there were any relevant networking changes in that window.

So I think the next step is to ask you to continue with (or redo) that
bisection search.

Is there someone who is more git-competent than I who could help with
this please?