regression caused by: genirq: do not leave interupts enabled on free_irq

regression caused by: genirq: do not leave interupts enabled on free_irq

[Yinghai Lu]

last week found:
after latest kernel kexec RHEL 5.1 or other stack kernel, the nvidia
forcedeth doesn't work anymore.

I stared at forcedeth.c two days. and revert every patches about that
doesn't help.

and figure out 2.6.25-rc2 works.

with git-bisect found

commit 89d694b9dbe769ca1004e01db0ca43964806a611
Author: Thomas Gleixner <tglx@linutronix.de>
Date:   Mon Feb 18 18:25:17 2008 +0100

    genirq: do not leave interupts enabled on free_irq

    The default_disable() function was changed in commit:

     76d2160147f43f982dfe881404cfde9fd0a9da21
     genirq: do not mask interrupts by default

    It removed the mask function in favour of the default delayed
    interrupt disabling. Unfortunately this also broke the shutdown in
    free_irq() when the last handler is removed from the interrupt for
    those architectures which rely on the default implementations. Now we
    can end up with a enabled interrupt line after the last handler was
    removed, which can result in spurious interrupts.

    Fix this by adding a default_shutdown function, which is only
    installed, when the irqchip implementation does provide neither a
    shutdown nor a disable function.

    [@stable: affected versions: .21 - .24 ]

    Pointed-out-by: Michael Hennerich <Michael.Hennerich@analog.com>
    Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
    Acked-by: Ingo Molnar <mingo@elte.hu>
    Cc: stable@kernel.org
    Tested-by: Michael Hennerich <Michael.Hennerich@analog.com>

caused the regression.

it affected all my servers that are using Nvidia MCP55.

maybe we need to revert that patch.

YH

Re: regression caused by: genirq: do not leave interupts enabled on free_irq

[Thomas Gleixner]

On Thu, 10 Apr 2008, Yinghai Lu wrote:
> last week found:
> after latest kernel kexec RHEL 5.1 or other stack kernel, the nvidia
> forcedeth doesn't work anymore.
> 
> I stared at forcedeth.c two days. and revert every patches about that
> doesn't help.

So forcedeth does not come up again, when you kexec from linus.git
into an older distro kernel. Or is it the other way round ?

Does a non kexec boot work ?

> and figure out 2.6.25-rc2 works.
> 
> with git-bisect found
> 
> commit 89d694b9dbe769ca1004e01db0ca43964806a611
> Author: Thomas Gleixner <tglx@linutronix.de>
> Date:   Mon Feb 18 18:25:17 2008 +0100
> 
>     genirq: do not leave interupts enabled on free_irq
> 
> caused the regression.
> 
> it affected all my servers that are using Nvidia MCP55.
> 
> maybe we need to revert that patch.

Which causes an interrupt storm on the enabled irq line, when the
interrupt line is still active for whatever reason. So we trade one
badness vs. the other.

Reverting the patch is not going to give us any answer about the real
problem.

Is there anything in dmesg, which might give us an hint about that ?

Thanks,
	tglx

Re: regression caused by: genirq: do not leave interupts enabled on free_irq

[Yinghai Lu]

On Fri, Apr 11, 2008 at 12:01 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
> On Thu, 10 Apr 2008, Yinghai Lu wrote:
>  > last week found:
>  > after latest kernel kexec RHEL 5.1 or other stack kernel, the nvidia
>  > forcedeth doesn't work anymore.
>  >
>  > I stared at forcedeth.c two days. and revert every patches about that
>  > doesn't help.
>
>  So forcedeth does not come up again, when you kexec from linus.git
>  into an older distro kernel. Or is it the other way round ?
RHEL 5.1 kexec RHEL 5.1 : works
RHEL 5.1 kexec linus kernel: works
linus (after -rc2) kexec linus tree: works:
linus  (after -rc2) kexec RHEL 5.1 : forcedeth will not come up
linus ( before -rc2 include rc2) kexec RHEL 5.1 works

>
>  Does a non kexec boot work ?

Yes

>
>
>  > and figure out 2.6.25-rc2 works.
>  >
>  > with git-bisect found
>  >
>  > commit 89d694b9dbe769ca1004e01db0ca43964806a611
>  > Author: Thomas Gleixner <tglx@linutronix.de>
>  > Date:   Mon Feb 18 18:25:17 2008 +0100
>  >
>  >     genirq: do not leave interupts enabled on free_irq
>  >
>
> > caused the regression.
>  >
>  > it affected all my servers that are using Nvidia MCP55.
>  >
>  > maybe we need to revert that patch.
>
>  Which causes an interrupt storm on the enabled irq line, when the
>  interrupt line is still active for whatever reason. So we trade one
>  badness vs. the other.
>
>  Reverting the patch is not going to give us any answer about the real
>  problem.

but that interrupt storm bug already happend from .21 to .24

>
>  Is there anything in dmesg, which might give us an hint about that ?

everything looks normal.

YH

Re: regression caused by: genirq: do not leave interupts enabled on free_irq

[Yinghai Lu]

On Fri, Apr 11, 2008 at 12:14 AM, Yinghai Lu <yhlu.kernel@gmail.com> wrote:
> On Fri, Apr 11, 2008 at 12:01 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
>  > On Thu, 10 Apr 2008, Yinghai Lu wrote:
>  >  > last week found:
>  >  > after latest kernel kexec RHEL 5.1 or other stack kernel, the nvidia
>  >  > forcedeth doesn't work anymore.
>  >  >
>  >  > I stared at forcedeth.c two days. and revert every patches about that
>  >  > doesn't help.
>  >
>  >  So forcedeth does not come up again, when you kexec from linus.git
>  >  into an older distro kernel. Or is it the other way round ?
>  RHEL 5.1 kexec RHEL 5.1 : works
>  RHEL 5.1 kexec linus kernel: works
>  linus (after -rc2) kexec linus tree: works:
>  linus  (after -rc2) kexec RHEL 5.1 : forcedeth will not come up
>  linus ( before -rc2 include rc2) kexec RHEL 5.1 works
>
the forcedeth can not get IP address...

YH

Re: regression caused by: genirq: do not leave interupts enabled on free_irq

[Eric W. Biederman]

On Fri, 2008-04-11 at 00:17 -0700, Yinghai Lu wrote:
> On Fri, Apr 11, 2008 at 12:14 AM, Yinghai Lu <yhlu.kernel@gmail.com> wrote:
> > On Fri, Apr 11, 2008 at 12:01 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
> >  > On Thu, 10 Apr 2008, Yinghai Lu wrote:
> >  >  > last week found:
> >  >  > after latest kernel kexec RHEL 5.1 or other stack kernel, the nvidia
> >  >  > forcedeth doesn't work anymore.
> >  >  >
> >  >  > I stared at forcedeth.c two days. and revert every patches about that
> >  >  > doesn't help.
> >  >
> >  >  So forcedeth does not come up again, when you kexec from linus.git
> >  >  into an older distro kernel. Or is it the other way round ?
> >  RHEL 5.1 kexec RHEL 5.1 : works
> >  RHEL 5.1 kexec linus kernel: works
> >  linus (after -rc2) kexec linus tree: works:
> >  linus  (after -rc2) kexec RHEL 5.1 : forcedeth will not come up
> >  linus ( before -rc2 include rc2) kexec RHEL 5.1 works
> >
> the forcedeth can not get IP address...

Sounds like you are not getting any interrupts when you receive a
packet.  (i.e. The interrupt line is staying disabled).

Is MSI an option here?  I'm wondering if we disable the MSI and
something is not enabling it.

Eric

Re: regression caused by: genirq: do not leave interupts enabled on free_irq

[Yinghai Lu]

On Fri, Apr 11, 2008 at 12:30 AM, Eric W. Biederman
<ebiederm@xmission.com> wrote:
>
>
>  On Fri, 2008-04-11 at 00:17 -0700, Yinghai Lu wrote:
>  > On Fri, Apr 11, 2008 at 12:14 AM, Yinghai Lu <yhlu.kernel@gmail.com> wrote:
>  > > On Fri, Apr 11, 2008 at 12:01 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
>  > >  > On Thu, 10 Apr 2008, Yinghai Lu wrote:
>  > >  >  > last week found:
>  > >  >  > after latest kernel kexec RHEL 5.1 or other stack kernel, the nvidia
>  > >  >  > forcedeth doesn't work anymore.
>  > >  >  >
>  > >  >  > I stared at forcedeth.c two days. and revert every patches about that
>  > >  >  > doesn't help.
>  > >  >
>  > >  >  So forcedeth does not come up again, when you kexec from linus.git
>  > >  >  into an older distro kernel. Or is it the other way round ?
>  > >  RHEL 5.1 kexec RHEL 5.1 : works
>  > >  RHEL 5.1 kexec linus kernel: works
>  > >  linus (after -rc2) kexec linus tree: works:
>  > >  linus  (after -rc2) kexec RHEL 5.1 : forcedeth will not come up
>  > >  linus ( before -rc2 include rc2) kexec RHEL 5.1 works
>  > >
>  > the forcedeth can not get IP address...
>
>  Sounds like you are not getting any interrupts when you receive a
>  packet.  (i.e. The interrupt line is staying disabled).
>
>  Is MSI an option here?  I'm wondering if we disable the MSI and
>  something is not enabling it.

mcp55 is using msi for forcedeth

MSI.. will look at the driver in RHEL 5.1

it seems that another server with ck804 works because it is using ioapic.

YH

Re: regression caused by: genirq: do not leave interupts enabled on free_irq

[Thomas Gleixner]

On Fri, 11 Apr 2008, Eric W. Biederman wrote:
> On Fri, 2008-04-11 at 00:17 -0700, Yinghai Lu wrote:
> > On Fri, Apr 11, 2008 at 12:14 AM, Yinghai Lu <yhlu.kernel@gmail.com> wrote:
> > > On Fri, Apr 11, 2008 at 12:01 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
> > >  > On Thu, 10 Apr 2008, Yinghai Lu wrote:
> > >  >  > last week found:
> > >  >  > after latest kernel kexec RHEL 5.1 or other stack kernel, the nvidia
> > >  >  > forcedeth doesn't work anymore.
> > >  >  >
> > >  >  > I stared at forcedeth.c two days. and revert every patches about that
> > >  >  > doesn't help.
> > >  >
> > >  >  So forcedeth does not come up again, when you kexec from linus.git
> > >  >  into an older distro kernel. Or is it the other way round ?
> > >  RHEL 5.1 kexec RHEL 5.1 : works
> > >  RHEL 5.1 kexec linus kernel: works
> > >  linus (after -rc2) kexec linus tree: works:
> > >  linus  (after -rc2) kexec RHEL 5.1 : forcedeth will not come up
> > >  linus ( before -rc2 include rc2) kexec RHEL 5.1 works
> > >
> > the forcedeth can not get IP address...
> 
> Sounds like you are not getting any interrupts when you receive a
> packet.  (i.e. The interrupt line is staying disabled).
> 
> Is MSI an option here?  I'm wondering if we disable the MSI and
> something is not enabling it.

Hmm, we disable the interrupt on free_irq(), but we reenable it in
request_irq()/setup_irq(), which is called when the forcedeth driver
initializes in the kexeced kernel. So there is some other deeper down
problem lurking.

Yinghai, can you apply that patch to RHEL 5.1 and check, what happens
if you do:

modprobe forcedeth
ifup ...
ifdown ...
rmmod forcedeth
modprobe forcedeth
ifup ...

This should result in the same problem, but probably simpler to debug.

Thanks,
	tglx

Re: regression caused by: genirq: do not leave interupts enabled on free_irq

[Rafael J. Wysocki]

On Friday, 11 of April 2008, Thomas Gleixner wrote:
> On Fri, 11 Apr 2008, Eric W. Biederman wrote:
> > On Fri, 2008-04-11 at 00:17 -0700, Yinghai Lu wrote:
> > > On Fri, Apr 11, 2008 at 12:14 AM, Yinghai Lu <yhlu.kernel@gmail.com> wrote:
> > > > On Fri, Apr 11, 2008 at 12:01 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
> > > >  > On Thu, 10 Apr 2008, Yinghai Lu wrote:
> > > >  >  > last week found:
> > > >  >  > after latest kernel kexec RHEL 5.1 or other stack kernel, the nvidia
> > > >  >  > forcedeth doesn't work anymore.
> > > >  >  >
> > > >  >  > I stared at forcedeth.c two days. and revert every patches about that
> > > >  >  > doesn't help.
> > > >  >
> > > >  >  So forcedeth does not come up again, when you kexec from linus.git
> > > >  >  into an older distro kernel. Or is it the other way round ?
> > > >  RHEL 5.1 kexec RHEL 5.1 : works
> > > >  RHEL 5.1 kexec linus kernel: works
> > > >  linus (after -rc2) kexec linus tree: works:
> > > >  linus  (after -rc2) kexec RHEL 5.1 : forcedeth will not come up
> > > >  linus ( before -rc2 include rc2) kexec RHEL 5.1 works
> > > >
> > > the forcedeth can not get IP address...
> > 
> > Sounds like you are not getting any interrupts when you receive a
> > packet.  (i.e. The interrupt line is staying disabled).
> > 
> > Is MSI an option here?  I'm wondering if we disable the MSI and
> > something is not enabling it.
> 
> Hmm, we disable the interrupt on free_irq(), but we reenable it in
> request_irq()/setup_irq(), which is called when the forcedeth driver
> initializes in the kexeced kernel. So there is some other deeper down
> problem lurking.
> 
> Yinghai, can you apply that patch to RHEL 5.1 and check, what happens
> if you do:
> 
> modprobe forcedeth
> ifup ...
> ifdown ...
> rmmod forcedeth
> modprobe forcedeth
> ifup ...
> 
> This should result in the same problem, but probably simpler to debug.

I've just got a report from a forcedeth user that it doesn't work after a
resume from RAM for him any more.  Can it be related?

Rafael

Re: regression caused by: genirq: do not leave interupts enabled on free_irq

[Daniel Barkalow]

On Fri, 11 Apr 2008, Yinghai Lu wrote:

> On Fri, Apr 11, 2008 at 12:30 AM, Eric W. Biederman
> <ebiederm@xmission.com> wrote:
> >
> >
> >  On Fri, 2008-04-11 at 00:17 -0700, Yinghai Lu wrote:
> >  > On Fri, Apr 11, 2008 at 12:14 AM, Yinghai Lu <yhlu.kernel@gmail.com> wrote:
> >  > >  RHEL 5.1 kexec RHEL 5.1 : works
> >  > >  RHEL 5.1 kexec linus kernel: works
> >  > >  linus (after -rc2) kexec linus tree: works:
> >  > >  linus  (after -rc2) kexec RHEL 5.1 : forcedeth will not come up
> >  > >  linus ( before -rc2 include rc2) kexec RHEL 5.1 works
> >  > >
> >  > the forcedeth can not get IP address...
> >
> >  Sounds like you are not getting any interrupts when you receive a
> >  packet.  (i.e. The interrupt line is staying disabled).
> >
> >  Is MSI an option here?  I'm wondering if we disable the MSI and
> >  something is not enabling it.
> 
> mcp55 is using msi for forcedeth
> 
> MSI.. will look at the driver in RHEL 5.1
> 
> it seems that another server with ck804 works because it is using ioapic.

When turning on MSI for forcedeth, we do the INTx disable thing (and it's 
actually necessary for at least some forcedeth-driven hardware); if we're 
not disabling MSI (and making INTx undisabled), the old kernel may not 
undisable INTx but expect to have legacy interrupts delivered.

	-Daniel
*This .sig left intentionally blank*

Re: regression caused by: genirq: do not leave interupts enabled on free_irq

[Yinghai Lu]

On Fri, Apr 11, 2008 at 8:19 AM, Rafael J. Wysocki <rjw@sisk.pl> wrote:
>
> On Friday, 11 of April 2008, Thomas Gleixner wrote:
>  > On Fri, 11 Apr 2008, Eric W. Biederman wrote:
>  > > On Fri, 2008-04-11 at 00:17 -0700, Yinghai Lu wrote:
>  > > > On Fri, Apr 11, 2008 at 12:14 AM, Yinghai Lu <yhlu.kernel@gmail.com> wrote:
>  > > > > On Fri, Apr 11, 2008 at 12:01 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
>  > > > >  > On Thu, 10 Apr 2008, Yinghai Lu wrote:
>  > > > >  >  > last week found:
>  > > > >  >  > after latest kernel kexec RHEL 5.1 or other stack kernel, the nvidia
>  > > > >  >  > forcedeth doesn't work anymore.
>  > > > >  >  >
>  > > > >  >  > I stared at forcedeth.c two days. and revert every patches about that
>  > > > >  >  > doesn't help.
>  > > > >  >
>  > > > >  >  So forcedeth does not come up again, when you kexec from linus.git
>  > > > >  >  into an older distro kernel. Or is it the other way round ?
>  > > > >  RHEL 5.1 kexec RHEL 5.1 : works
>  > > > >  RHEL 5.1 kexec linus kernel: works
>  > > > >  linus (after -rc2) kexec linus tree: works:
>  > > > >  linus  (after -rc2) kexec RHEL 5.1 : forcedeth will not come up
>  > > > >  linus ( before -rc2 include rc2) kexec RHEL 5.1 works
>  > > > >
>  > > > the forcedeth can not get IP address...
>  > >
>  > > Sounds like you are not getting any interrupts when you receive a
>  > > packet.  (i.e. The interrupt line is staying disabled).
>  > >
>  > > Is MSI an option here?  I'm wondering if we disable the MSI and
>  > > something is not enabling it.
>  >
>  > Hmm, we disable the interrupt on free_irq(), but we reenable it in
>  > request_irq()/setup_irq(), which is called when the forcedeth driver
>  > initializes in the kexeced kernel. So there is some other deeper down
>  > problem lurking.
>  >
>  > Yinghai, can you apply that patch to RHEL 5.1 and check, what happens
>  > if you do:
>  >
>  > modprobe forcedeth
>  > ifup ...
>  > ifdown ...
>  > rmmod forcedeth
>  > modprobe forcedeth
>  > ifup ...
>  >
>  > This should result in the same problem, but probably simpler to debug.
>
>  I've just got a report from a forcedeth user that it doesn't work after a
>  resume from RAM for him any more.  Can it be related?

then another path that missing enable irq..

YH

Re: regression caused by: genirq: do not leave interupts enabled on free_irq

[Yinghai Lu]

[-- Attachment #1: Type: text/plain, Size: 1440 bytes --]

On Fri, Apr 11, 2008 at 12:30 AM, Eric W. Biederman
<ebiederm@xmission.com> wrote:
>
>
>  On Fri, 2008-04-11 at 00:17 -0700, Yinghai Lu wrote:
>  > On Fri, Apr 11, 2008 at 12:14 AM, Yinghai Lu <yhlu.kernel@gmail.com> wrote:
>  > > On Fri, Apr 11, 2008 at 12:01 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
>  > >  > On Thu, 10 Apr 2008, Yinghai Lu wrote:
>  > >  >  > last week found:
>  > >  >  > after latest kernel kexec RHEL 5.1 or other stack kernel, the nvidia
>  > >  >  > forcedeth doesn't work anymore.
>  > >  >  >
>  > >  >  > I stared at forcedeth.c two days. and revert every patches about that
>  > >  >  > doesn't help.
>  > >  >
>  > >  >  So forcedeth does not come up again, when you kexec from linus.git
>  > >  >  into an older distro kernel. Or is it the other way round ?
>  > >  RHEL 5.1 kexec RHEL 5.1 : works
>  > >  RHEL 5.1 kexec linus kernel: works
>  > >  linus (after -rc2) kexec linus tree: works:
>  > >  linus  (after -rc2) kexec RHEL 5.1 : forcedeth will not come up
>  > >  linus ( before -rc2 include rc2) kexec RHEL 5.1 works
>  > >
>  > the forcedeth can not get IP address...
>
>  Sounds like you are not getting any interrupts when you receive a
>  packet.  (i.e. The interrupt line is staying disabled).
>
>  Is MSI an option here?  I'm wondering if we disable the MSI and
>  something is not enabling it.



after
setpci -s 0x00:08.0 0x60.b=0xfe

the eth0 can get ip address with ifup eth0.

YH

[-- Attachment #2: not_works.txt --]
[-- Type: text/plain, Size: 1896 bytes --]

00:08.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a3)
	Subsystem: nVidia Corporation Unknown device cb84
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0 (250ns min, 5000ns max)
	Interrupt: pin A routed to IRQ 233
	Region 0: Memory at defba000 (32-bit, non-prefetchable) [size=4K]
	Region 1: I/O ports at 8400 [size=8]
	Region 2: Memory at defbe800 (32-bit, non-prefetchable) [size=256]
	Region 3: Memory at defbe400 (32-bit, non-prefetchable) [size=16]
	Capabilities: [44] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 PME-Enable+ DSel=0 DScale=0 PME-
	Capabilities: [70] MSI-X: Enable- Mask- TabSize=8
		Vector table: BAR=2 offset=00000000
		PBA: BAR=3 offset=00000000
	Capabilities: [50] Message Signalled Interrupts: 64bit+ Queue=0/3 Enable-
		Address: 00000000fee04000  Data: 406a
	Capabilities: [6c] HyperTransport: MSI Mapping
00: de 10 73 03 07 00 b0 00 a3 00 80 06 00 00 00 00
10: 00 a0 fb de 01 84 00 00 00 e8 fb de 00 e4 fb de
20: 00 00 00 00 00 00 00 00 00 00 00 00 de 10 84 cb
30: 00 00 00 00 44 00 00 00 00 00 00 00 0a 01 01 14
40: de 10 84 cb 01 70 02 fe 00 01 00 00 0c 00 00 00
50: 05 6c 86 01 00 40 e0 fe 00 00 00 00 6a 40 00 00
60: ff 00 00 00 00 00 00 00 ff 00 00 00 08 00 03 a8
70: 11 50 07 00 02 00 00 00 03 00 00 00 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 11 00 00 00 52 01 00 00 00 00 00 00


[-- Attachment #3: works.txt --]
[-- Type: text/plain, Size: 1896 bytes --]

00:08.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a3)
	Subsystem: nVidia Corporation Unknown device cb84
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0 (250ns min, 5000ns max)
	Interrupt: pin A routed to IRQ 106
	Region 0: Memory at defba000 (32-bit, non-prefetchable) [size=4K]
	Region 1: I/O ports at 8400 [size=8]
	Region 2: Memory at defbe800 (32-bit, non-prefetchable) [size=256]
	Region 3: Memory at defbe400 (32-bit, non-prefetchable) [size=16]
	Capabilities: [44] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 PME-Enable+ DSel=0 DScale=0 PME-
	Capabilities: [70] MSI-X: Enable- Mask- TabSize=8
		Vector table: BAR=2 offset=00000000
		PBA: BAR=3 offset=00000000
	Capabilities: [50] Message Signalled Interrupts: 64bit+ Queue=0/3 Enable+
		Address: 00000000fee07000  Data: 406a
	Capabilities: [6c] HyperTransport: MSI Mapping
00: de 10 73 03 07 00 b0 00 a3 00 80 06 00 00 00 00
10: 00 a0 fb de 01 84 00 00 00 e8 fb de 00 e4 fb de
20: 00 00 00 00 00 00 00 00 00 00 00 00 de 10 84 cb
30: 00 00 00 00 44 00 00 00 00 00 00 00 0a 01 01 14
40: de 10 84 cb 01 70 02 fe 00 01 00 00 0c 00 00 00
50: 05 6c 87 01 00 70 e0 fe 00 00 00 00 6a 40 00 00
60: fe 00 00 00 00 00 00 00 ff 00 00 00 08 00 03 a8
70: 11 50 07 00 02 00 00 00 03 00 00 00 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 11 00 00 00 52 01 00 00 00 00 00 00

Re: regression caused by: genirq: do not leave interupts enabled on free_irq

[Yinghai Lu]

On Fri, Apr 11, 2008 at 11:22 AM, Yinghai Lu <yhlu.kernel@gmail.com> wrote:
> On Fri, Apr 11, 2008 at 12:30 AM, Eric W. Biederman
>  <ebiederm@xmission.com> wrote:
>  >
>  >
>
>
> >  On Fri, 2008-04-11 at 00:17 -0700, Yinghai Lu wrote:
>  >  > On Fri, Apr 11, 2008 at 12:14 AM, Yinghai Lu <yhlu.kernel@gmail.com> wrote:
>  >  > > On Fri, Apr 11, 2008 at 12:01 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
>  >  > >  > On Thu, 10 Apr 2008, Yinghai Lu wrote:
>  >  > >  >  > last week found:
>  >  > >  >  > after latest kernel kexec RHEL 5.1 or other stack kernel, the nvidia
>  >  > >  >  > forcedeth doesn't work anymore.
>  >  > >  >  >
>  >  > >  >  > I stared at forcedeth.c two days. and revert every patches about that
>  >  > >  >  > doesn't help.
>  >  > >  >
>  >  > >  >  So forcedeth does not come up again, when you kexec from linus.git
>  >  > >  >  into an older distro kernel. Or is it the other way round ?
>  >  > >  RHEL 5.1 kexec RHEL 5.1 : works
>  >  > >  RHEL 5.1 kexec linus kernel: works
>  >  > >  linus (after -rc2) kexec linus tree: works:
>  >  > >  linus  (after -rc2) kexec RHEL 5.1 : forcedeth will not come up
>  >  > >  linus ( before -rc2 include rc2) kexec RHEL 5.1 works
>  >  > >
>  >  > the forcedeth can not get IP address...
>  >
>  >  Sounds like you are not getting any interrupts when you receive a
>  >  packet.  (i.e. The interrupt line is staying disabled).
>  >
>  >  Is MSI an option here?  I'm wondering if we disable the MSI and
>  >  something is not enabling it.
>
>
>
>  after
>  setpci -s 0x00:08.0 0x60.b=0xfe
>
>  the eth0 can get ip address with ifup eth0.

it is RHEL 5.1 problem...with msi.

somehow msi entry msi_attrib_maskbit=is_mask_bit_support always get
0...., and use msi_irq_wo_maskbit_type instead of
msi_irq_w_maskbit_type

so when load that directly without kexec
that 0x60 always to 0x00.

and this time kernel after 2.6.25-rc2 ( not included) really shutdown
the msi with 0xff the all maskbit...

YH

Re: regression caused by: genirq: do not leave interupts enabled on free_irq

[Andrew Morton]

On Fri, 11 Apr 2008 12:48:43 -0700 "Yinghai Lu" <yhlu.kernel@gmail.com> wrote:

> On Fri, Apr 11, 2008 at 11:22 AM, Yinghai Lu <yhlu.kernel@gmail.com> wrote:
> > On Fri, Apr 11, 2008 at 12:30 AM, Eric W. Biederman
> >  <ebiederm@xmission.com> wrote:
> >  >
> >  >
> >
> >
> > >  On Fri, 2008-04-11 at 00:17 -0700, Yinghai Lu wrote:
> >  >  > On Fri, Apr 11, 2008 at 12:14 AM, Yinghai Lu <yhlu.kernel@gmail.com> wrote:
> >  >  > > On Fri, Apr 11, 2008 at 12:01 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
> >  >  > >  > On Thu, 10 Apr 2008, Yinghai Lu wrote:
> >  >  > >  >  > last week found:
> >  >  > >  >  > after latest kernel kexec RHEL 5.1 or other stack kernel, the nvidia
> >  >  > >  >  > forcedeth doesn't work anymore.
> >  >  > >  >  >
> >  >  > >  >  > I stared at forcedeth.c two days. and revert every patches about that
> >  >  > >  >  > doesn't help.
> >  >  > >  >
> >  >  > >  >  So forcedeth does not come up again, when you kexec from linus.git
> >  >  > >  >  into an older distro kernel. Or is it the other way round ?
> >  >  > >  RHEL 5.1 kexec RHEL 5.1 : works
> >  >  > >  RHEL 5.1 kexec linus kernel: works
> >  >  > >  linus (after -rc2) kexec linus tree: works:
> >  >  > >  linus  (after -rc2) kexec RHEL 5.1 : forcedeth will not come up
> >  >  > >  linus ( before -rc2 include rc2) kexec RHEL 5.1 works
> >  >  > >
> >  >  > the forcedeth can not get IP address...
> >  >
> >  >  Sounds like you are not getting any interrupts when you receive a
> >  >  packet.  (i.e. The interrupt line is staying disabled).
> >  >
> >  >  Is MSI an option here?  I'm wondering if we disable the MSI and
> >  >  something is not enabling it.
> >
> >
> >
> >  after
> >  setpci -s 0x00:08.0 0x60.b=0xfe
> >
> >  the eth0 can get ip address with ifup eth0.
> 
> it is RHEL 5.1 problem...with msi.
> 
> somehow msi entry msi_attrib_maskbit=is_mask_bit_support always get
> 0...., and use msi_irq_wo_maskbit_type instead of
> msi_irq_w_maskbit_type
> 
> so when load that directly without kexec
> that 0x60 always to 0x00.
> 
> and this time kernel after 2.6.25-rc2 ( not included) really shutdown
> the msi with 0xff the all maskbit...
> 

So... where did this end up?   It's a post-2.6.24 regression, isn't it?

Do we think it's a bug in the RHEL 5.1 kernel?  Even so, that's a problem
for the 2.6.25 kernel.

Re: regression caused by: genirq: do not leave interupts enabled on free_irq

[Yinghai Lu]

On Tue, Apr 15, 2008 at 12:44 AM, Andrew Morton
<akpm@linux-foundation.org> wrote:
>
> On Fri, 11 Apr 2008 12:48:43 -0700 "Yinghai Lu" <yhlu.kernel@gmail.com> wrote:
>
>  > On Fri, Apr 11, 2008 at 11:22 AM, Yinghai Lu <yhlu.kernel@gmail.com> wrote:
>  > > On Fri, Apr 11, 2008 at 12:30 AM, Eric W. Biederman
>  > >  <ebiederm@xmission.com> wrote:
>  > >  >
>  > >  >
>  > >
>  > >
>  > > >  On Fri, 2008-04-11 at 00:17 -0700, Yinghai Lu wrote:
>  > >  >  > On Fri, Apr 11, 2008 at 12:14 AM, Yinghai Lu <yhlu.kernel@gmail.com> wrote:
>  > >  >  > > On Fri, Apr 11, 2008 at 12:01 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
>  > >  >  > >  > On Thu, 10 Apr 2008, Yinghai Lu wrote:
>  > >  >  > >  >  > last week found:
>  > >  >  > >  >  > after latest kernel kexec RHEL 5.1 or other stack kernel, the nvidia
>  > >  >  > >  >  > forcedeth doesn't work anymore.
>  > >  >  > >  >  >
>  > >  >  > >  >  > I stared at forcedeth.c two days. and revert every patches about that
>  > >  >  > >  >  > doesn't help.
>  > >  >  > >  >
>  > >  >  > >  >  So forcedeth does not come up again, when you kexec from linus.git
>  > >  >  > >  >  into an older distro kernel. Or is it the other way round ?
>  > >  >  > >  RHEL 5.1 kexec RHEL 5.1 : works
>  > >  >  > >  RHEL 5.1 kexec linus kernel: works
>  > >  >  > >  linus (after -rc2) kexec linus tree: works:
>  > >  >  > >  linus  (after -rc2) kexec RHEL 5.1 : forcedeth will not come up
>  > >  >  > >  linus ( before -rc2 include rc2) kexec RHEL 5.1 works
>  > >  >  > >
>  > >  >  > the forcedeth can not get IP address...
>  > >  >
>  > >  >  Sounds like you are not getting any interrupts when you receive a
>  > >  >  packet.  (i.e. The interrupt line is staying disabled).
>  > >  >
>  > >  >  Is MSI an option here?  I'm wondering if we disable the MSI and
>  > >  >  something is not enabling it.
>  > >
>  > >
>  > >
>  > >  after
>  > >  setpci -s 0x00:08.0 0x60.b=0xfe
>  > >
>  > >  the eth0 can get ip address with ifup eth0.
>  >
>  > it is RHEL 5.1 problem...with msi.
>  >
>  > somehow msi entry msi_attrib_maskbit=is_mask_bit_support always get
>  > 0...., and use msi_irq_wo_maskbit_type instead of
>  > msi_irq_w_maskbit_type
>  >
>  > so when load that directly without kexec
>  > that 0x60 always to 0x00.
>  >
>  > and this time kernel after 2.6.25-rc2 ( not included) really shutdown
>  > the msi with 0xff the all maskbit...
>  >
>
>  So... where did this end up?   It's a post-2.6.24 regression, isn't it?
>
>  Do we think it's a bug in the RHEL 5.1 kernel?  Even so, that's a problem
>  for the 2.6.25 kernel.

it is RHEL 5.1 kernel problem, it has code for maskbit with msi there,
but that does work...

I sent out patch for 2.6.25 so it can be used to kexec old kernel that
doesn't support maskbit for MSI.

please comment...

[PATCH] x86_64: restore mask_bits in msi shutdown


for MSI, default_shutdown will call mask_bit for msi device. so all
mask bits will
left disabled after free_irq.
then if kexec next kernel that only can use msi_enable bit.
all device's MSI can not be used.

So try to restore MSI mask bits that is saved before using msi in first kernel.

http://lkml.org/lkml/2008/4/11/369

YH

Re: regression caused by: genirq: do not leave interupts enabled on free_irq

[Yinghai Lu]

On Tue, Apr 15, 2008 at 12:44 AM, Andrew Morton
<akpm@linux-foundation.org> wrote:
>
> On Fri, 11 Apr 2008 12:48:43 -0700 "Yinghai Lu" <yhlu.kernel@gmail.com> wrote:
>
>  > On Fri, Apr 11, 2008 at 11:22 AM, Yinghai Lu <yhlu.kernel@gmail.com> wrote:
>  > > On Fri, Apr 11, 2008 at 12:30 AM, Eric W. Biederman
>  > >  <ebiederm@xmission.com> wrote:
>  > >  >
>  > >  >
>  > >
>  > >
>  > > >  On Fri, 2008-04-11 at 00:17 -0700, Yinghai Lu wrote:
>  > >  >  > On Fri, Apr 11, 2008 at 12:14 AM, Yinghai Lu <yhlu.kernel@gmail.com> wrote:
>  > >  >  > > On Fri, Apr 11, 2008 at 12:01 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
>  > >  >  > >  > On Thu, 10 Apr 2008, Yinghai Lu wrote:
>  > >  >  > >  >  > last week found:
>  > >  >  > >  >  > after latest kernel kexec RHEL 5.1 or other stack kernel, the nvidia
>  > >  >  > >  >  > forcedeth doesn't work anymore.
>  > >  >  > >  >  >
>  > >  >  > >  >  > I stared at forcedeth.c two days. and revert every patches about that
>  > >  >  > >  >  > doesn't help.
>  > >  >  > >  >
>  > >  >  > >  >  So forcedeth does not come up again, when you kexec from linus.git
>  > >  >  > >  >  into an older distro kernel. Or is it the other way round ?
>  > >  >  > >  RHEL 5.1 kexec RHEL 5.1 : works
>  > >  >  > >  RHEL 5.1 kexec linus kernel: works
>  > >  >  > >  linus (after -rc2) kexec linus tree: works:
>  > >  >  > >  linus  (after -rc2) kexec RHEL 5.1 : forcedeth will not come up
>  > >  >  > >  linus ( before -rc2 include rc2) kexec RHEL 5.1 works
>  > >  >  > >
>  > >  >  > the forcedeth can not get IP address...
>  > >  >
>  > >  >  Sounds like you are not getting any interrupts when you receive a
>  > >  >  packet.  (i.e. The interrupt line is staying disabled).
>  > >  >
>  > >  >  Is MSI an option here?  I'm wondering if we disable the MSI and
>  > >  >  something is not enabling it.
>  > >
>  > >
>  > >
>  > >  after
>  > >  setpci -s 0x00:08.0 0x60.b=0xfe
>  > >
>  > >  the eth0 can get ip address with ifup eth0.
>  >
>  > it is RHEL 5.1 problem...with msi.
>  >
>  > somehow msi entry msi_attrib_maskbit=is_mask_bit_support always get
>  > 0...., and use msi_irq_wo_maskbit_type instead of
>  > msi_irq_w_maskbit_type
>  >
>  > so when load that directly without kexec
>  > that 0x60 always to 0x00.
>  >
>  > and this time kernel after 2.6.25-rc2 ( not included) really shutdown
>  > the msi with 0xff the all maskbit...
>  >
>
>  So... where did this end up?   It's a post-2.6.24 regression, isn't it?

between 2.6.25-rc2 and 2.6.25-rc3 by one patch from Thomas regarding
genirq to solve one long standing spurious problem...
looks like the irq is ioapic or pic, right?

YH

Re: regression caused by: genirq: do not leave interupts enabled on free_irq

[Ingo Molnar]

* Yinghai Lu <yhlu.kernel@gmail.com> wrote:

> >  Do we think it's a bug in the RHEL 5.1 kernel?  Even so, that's a 
> >  problem for the 2.6.25 kernel.
> 
> it is RHEL 5.1 kernel problem, it has code for maskbit with msi there, 
> but that does work...

addendum - basically the argument is: we should kexec conservatively - 
i.e. we should leave as few assumptions and hw changes around as 
possible. So restoring the MSI state is fair enough as a robustness fix, 
even though only kexec to older kernels is affected.

clearly 2.6.26 material, not 2.6.25.

	Ingo

Re: regression caused by: genirq: do not leave interupts enabled on free_irq

[Yinghai Lu]

On Tue, Apr 15, 2008 at 1:17 AM, Ingo Molnar <mingo@elte.hu> wrote:
>
>  * Yinghai Lu <yhlu.kernel@gmail.com> wrote:
>
>
> > >  Do we think it's a bug in the RHEL 5.1 kernel?  Even so, that's a
>  > >  problem for the 2.6.25 kernel.
>  >
>  > it is RHEL 5.1 kernel problem, it has code for maskbit with msi there,
>  > but that does work...
typo: should be "that does *not* work"
>
>  addendum - basically the argument is: we should kexec conservatively -
>  i.e. we should leave as few assumptions and hw changes around as
>  possible. So restoring the MSI state is fair enough as a robustness fix,
>  even though only kexec to older kernels is affected.

i keep using kexec back and forth to avoid BIOS post stage.

YH