Re: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slow

Martin Knoblauch
2009-05-27T11:26:13Z

[-- Attachment #1: Type: text/plain, Size: 5358 bytes --]

----- Original Message ----

> From: Martin Knoblauch <knobi@knobisoft.de>
> To: Andrew Morton <akpm@linux-foundation.org>
> Cc: Mike Galbraith <efault@gmx.de>; viro@ZenIV.linux.org.uk; rjw@sisk.pl; linux-kernel@vger.kernel.org; tigran@aivazian.fsnet.co.uks; Kay Sievers <kay.sievers@vrfy.org>; shemminger@vyatta.com; Jesse Barnes <jbarnes@virtuousgeek.org>; Matthew Wilcox <matthew@wil.cx>
> Sent: Wednesday, May 27, 2009 11:14:06 AM
> Subject: Re: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slow
>
> ----- Original Message ----
>
> > From: Andrew Morton
> > To: Martin Knoblauch
> > Cc: Mike Galbraith ; viro@ZenIV.linux.org.uk; rjw@sisk.pl;
> linux-kernel@vger.kernel.org; tigran@aivazian.fsnet.co.uks; Kay Sievers
> ; shemminger@vyatta.com; Jesse Barnes
> ; Matthew Wilcox
> > Sent: Wednesday, May 27, 2009 8:31:02 AM
> > Subject: Re: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slow
> >
> > On Wed, 20 May 2009 03:22:28 -0700 (PDT) Martin Knoblauch
> > wrote:
> >
> > >
> > > ----- Original Message ----
> > >
> > > > From: Mike Galbraith
> > > > To: Martin Knoblauch
> > > > Cc: Andrew Morton ; viro@ZenIV.linux.org.uk;
> > rjw@sisk.pl; linux-kernel@vger.kernel.org; tigran@aivazian.fsnet.co.uk
> > > > Sent: Wednesday, May 6, 2009 10:37:45 AM
> > > > Subject: Re: Analyzed/Solved: Booting 2.6.30-rc2-git7 very slow
> > > >
> > > > On Wed, 2009-05-06 at 00:55 -0700, Martin Knoblauch wrote:
> > > >
> > > > > just to bring this back to my problem :-)
> > > >
> > > > Good idea :-)
> > > >
> > > > > Last week I reported that the "new" sysfs entry in /proc/mounts already
>
> > comes
> > > > out of initrd. Does this ring a bell?
> > > > >
> > > > > http://lkml.indiana.edu/hypermail/linux/kernel/0904.3/03048.html
> > > >
> > > > Nope, no bells.
> > > >
> > > > The only thing I can suggest is that you try a bisection.
> > > >
> > > > -Mike
> > >
> > > OK, so I finally managed to bisect the issue down to the following commit.
> > Not much that I can say about it. Someone else suggested that it might all be
> a
> > question of timing. Might very well be. I will try it out on a system with a
> > different SCSI/RAID controller. The failing system has an "Smart Array 6i"
> > (cciss). "cciss", "ext3" and "jbd" are all modules coming from initrd.
> > >
> > > |commit 1120f8b8169fb2cb51219d326892d963e762edb6
> > > |Author: Stephen Hemminger
> > > |Date: Thu Dec 18 09:17:16 2008 -0800
> > > |
> > > | PCI: handle long delays in VPD access
> > > |
> > > | Accessing the VPD area can take a long time. The existing
> > > | VPD access code fails consistently on my hardware. There are comments
> > > |
> > > | Change the access routines to:
> > > | * use a mutex rather than spinning with IRQ's disabled and lock held
> > > | * have a much longer timeout
> > > | * call cond_resched while spinning
> > > |
> > > | Signed-off-by: Stephen Hemminger
> > > | Reviewed-by: Matthew Wilcox
> > > | Signed-off-by: Jesse Barnes
> > >
> >
> >
> >
> > So afacit what's happening is that the above change caused one of your
> > PCI devices to take a very long time to initialise, yes? Was it the
> > CCISS driver?
> >
>
> the whole thing is not understood. I mentioned CCISS only because it is the most
> visible difference difference between my two test platforms.
>
> > If you add "printk.time=y" to the kernel boot command line then you'll
> > get timestamped boot messages which will make it easier to determine
> > where the time was consumed. Adding `initcall_debug' to the boot line
> > will help us delve further into the delay, assuming that the offending
> > driver is build into vmlinux (which it might not be).
> >
>
> added both options. "dmesg" output from both is appended. The initcall timings
> vary in both directions. For CCISS, they are actually faster for the "bad" case.
>
> > Either way, it would be useful to know which driver the above change
> > broke.
> >
>
> agreed.
>
> > Once we know that, the questions is: doe sthe driver still work? If
> > so, then presumably the hardware if behaving unexpectedly, or in a way
> > which we're failing to cope with.
> >
>
> if it is CCISS, I can definitely say that it does work OK. As far as I can see,
> the whole system works OK, besides the duplicate sysfs line coming out of
> initrd.
>
> > Or perhaps that patch was simply buggy.
> >
> > btw, I don't agree that this report should be closed for "fuzziness"!
> > AFACIT the regression clearly and reproducibly occurs on one of your
> > machines, yes? That ain't fuzzy!
>
> frankly, I will stop caring about the DL380s before 2.6.31 gets released. My
> production kernels are not affected and the hotplug scripts can easily be fixed
> for testting. So, my interest is more curiosity. And the day-job does not really
> justify spending much more time on it. Aren't day-jobs annoying ...
>
> Cheers
> Martin

FWIW, I compiled the CCISS driver into the kernel. This makes the second "/sys" line in /proc/mounts go away, dmesg attached. But does it prove anything? The initialization of the CCISS hardware now happens about 2 seconds earlier in the bootup sequence. Does this hint to a problem with CCISS, or just confirms that the whole issue is really timing dependent? Anyway, I add Mike to CC.

Cheers
Martin

[-- Attachment #2: dmesg-g904d6a3-dirty-modscsi-nomodcciss-good --]
[-- Type: application/octet-stream, Size: 89578 bytes --]