Re: [Fastboot] Re: [RFC/PATCH] Kdump: Disabling PCI interrupts in capture kernel

Re: [Fastboot] Re: [RFC/PATCH] Kdump: Disabling PCI interrupts in capture kernel

[Maneesh Soni]

Quoting Greg KH <greg@kroah.com>:

> On Fri, Jun 03, 2005 at 04:55:24PM +0530, Vivek Goyal wrote:
> > Hi,
> > 
> > In kdump, sometimes, general driver initialization issues seems to be
> cropping 
> > in second kernel due to devices not being shutdown during crash and these
> 
> > devices are sending interrupts while second kernel is booting and drivers
> are 
> > not expecting any interrupts yet.
> 
> What are the errors you are seeing?
> How would the drivers be able to be getting interrupts delivered to them
> if they haven't registered the irq handler yet?

Probably the boot log with error messages could have been inlined instead
of putting as attachement. This is from attachement in Vievk's post


irq 11: nobody cared (try booting with the "irqpoll" option. 
 [<c103a3ef>] __report_bad_irq+0x2a/0x8d 
 [<c1039c28>] handle_IRQ_event+0x39/0x6d 
 [<c103a4eb>] note_interrupt+0x7f/0xe4 
 [<c1039dc5>] __do_IRQ+0x169/0x194 
 [<c1004cc6>] do_IRQ+0x1b/0x28 
 [<c1003256>] common_interrupt+0x1a/0x20 
 [<c101d61b>] __do_softirq+0x2b/0x88 
 [<c101d69e>] do_softirq+0x26/0x2a 
 [<c101d759>] irq_exit+0x33/0x35 
 [<c1004ccb>] do_IRQ+0x20/0x28 
 [<c1003256>] common_interrupt+0x1a/0x20 
 [<c1018916>] release_console_sem+0x43/0xf6 
 [<c101875f>] vprintk+0x16c/0x277 
 [<c1074dd8>] d_lookup+0x23/0x46 
 [<c10748ac>] d_alloc+0x136/0x1a6 
 [<c10185ef>] printk+0x17/0x1b 
 [<c120f33f>] hub_probe+0xa0/0x168 
 [<c1096dd1>] sysfs_new_dirent+0x1f/0x64 
 [<c120d37f>] usb_probe_interface+0x59/0x71 
 [<c116c9b2>] driver_probe_device+0x2f/0xa7 
 [<c116ca2a>] __device_attach+0x0/0x5 
 [<c116c259>] bus_for_each_drv+0x58/0x78 
 [<c116ca8f>] device_attach+0x60/0x64 
 [<c116ca2a>] __device_attach+0x0/0x5 
 [<c116c383>] bus_add_device+0x29/0xa6 
 [<c116f9c7>] device_pm_add+0x56/0x9c 
 [<c116b7c3>] device_add+0xc5/0x14a 
 [<c121557e>] usb_set_configuration+0x346/0x528 
 [<c120fa47>] usb_new_device+0xab/0x1be 
 [<c120007b>] ohci_iso_recv_set_channel_mask+0x3/0xc6 
 [<c12124e1>] register_root_hub+0xaf/0x162 
 [<c12133bf>] usb_add_hcd+0x172/0x396 
 [<c1217bce>] usb_hcd_pci_probe+0x26e/0x375 
 [<c10284e7>] __call_usermodehelper+0x0/0x61 
 [<c110cbe4>] pci_device_probe_static+0x40/0x54 
 [<c110cc27>] __pci_device_probe+0x2f/0x42 
 [<c110cc63>] pci_device_probe+0x29/0x47 
 [<c116c9b2>] driver_probe_device+0x2f/0xa7 
 [<c116ca93>] __driver_attach+0x0/0x43 
 [<c116cad4>] __driver_attach+0x41/0x43 
 [<c116c1c2>] bus_for_each_dev+0x5a/0x7a 
 [<c116cafc>] driver_attach+0x26/0x2a 
 [<c116ca93>] __driver_attach+0x0/0x43 
 [<c116c5cb>] bus_add_driver+0x6b/0xa5 
 [<c110ce97>] pci_register_driver+0x5e/0x7c 
 [<c14246f5>] init+0x1d/0x25 
 [<c140c7ed>] do_initcalls+0x54/0xb6 
 [<c100029c>] init+0x0/0x10c 
 [<c100029c>] init+0x0/0x10c 
 [<c10002c6>] init+0x2a/0x10c 
 [<c1001348>] kernel_thread_helper+0x0/0xb 
 [<c100134d>] kernel_thread_helper+0x5/0xb 
handlers: 
[<c11e3421>] (ahd_linux_isr+0x0/0x284) 
Disabling IRQ #11 


Thanks
Maneesh


-- 
Maneesh Soni
IBM Linux Technology Center
IBM India Software Labs,
Bangalore, India
Ph. 91-80-25044990
email: maneesh@in.ibm.com

Re: [Fastboot] Re: [RFC/PATCH] Kdump: Disabling PCI interrupts in capture kernel

[Dipankar Sarma]

On Sat, Jun 04, 2005 at 04:18:24PM +0300, Denis Vlasenko wrote:
> On Friday 03 June 2005 21:36, Eric W. Biederman wrote:
> > 
> > As I recall the drivers were not getting the interrupts but the interrupts
> > were happening.  To stop being spammed the kernel disables the irq line,
> > at the interrupt controller.  Then when the driver registered the
> > interrupt it would never receive the interrupt.
> 
> Shouldn't kernel keep all interrupt lines initially disabled 
> (sans platform-specific magic), and enable each like only when
> a device driver requests IRQ? This sounds simpler to do...

This doesn't help kdump folks. Interrupt pending from a device
from the first boot can flood the system when another driver
sharing the irq gets enabled in the second boot (kdump boot).
The disabling of interrupts need to be done on a per-device
basis for kdump to avoid this problem.

That said, I am not sure what is the issue with the console
drivers. What good is the irq for the console driver if
it hasn't requested for it ? Why should disabling it affect
consoles ? The interrupt will get enabled as soon as the driver
requests for it as per Vivek's patch. Am I missing something here ?

Thanks
Dipankar

Re: [Fastboot] Re: [RFC/PATCH] Kdump: Disabling PCI interrupts in capture kernel

[Dipankar Sarma]

On Sat, Jun 04, 2005 at 07:13:06PM +0530, Dipankar Sarma wrote:
> That said, I am not sure what is the issue with the console
> drivers. What good is the irq for the console driver if
> it hasn't requested for it ? Why should disabling it affect
> consoles ? The interrupt will get enabled as soon as the driver
> requests for it as per Vivek's patch. Am I missing something here ?

Doh! The answer is in earlier emails - fw controlled pci consoles.

Thanks
Dipankar

Re: [Fastboot] Re: [RFC/PATCH] Kdump: Disabling PCI interrupts in capture kernel

[Eric W. Biederman]

Dipankar Sarma <dipankar@in.ibm.com> writes:

> On Sat, Jun 04, 2005 at 07:13:06PM +0530, Dipankar Sarma wrote:
> > That said, I am not sure what is the issue with the console
> > drivers. What good is the irq for the console driver if
> > it hasn't requested for it ? Why should disabling it affect
> > consoles ? The interrupt will get enabled as soon as the driver
> > requests for it as per Vivek's patch. Am I missing something here ?
> 
> Doh! The answer is in earlier emails - fw controlled pci consoles.

There is still the question do fw controlled pci consoles
intersect with consoles used during kdump.  I would be
very surprised if the intersected but they might.

Eric

Re: [RFC/PATCH] Kdump: Disabling PCI interrupts in capture kernel

[Vivek Goyal]

Quoting Alan Stern <stern@rowland.harvard.edu>:

> On Sat, 4 Jun 2005, Vivek Goyal wrote:
> 
> > Hi Alan, I know very little about consoles and their working.
> > I had a question. Even if console is being managed by platform firmware,
> in
> > initial states of booting, does it require interrupts to be enabled at 
> > VGA contorller (at least for the simple text mode). I was quickly
> browsing
> > through drivers/video/console/vgacon.c and did not look like that this
> > console driver needed interrupts to be enabled at the controller.
> 
> This isn't an issue for VGA, as far as I know.  It applies to
> architectures like PPC-64 and perhaps Alpha or PA-Risc.  And I don't know
> the details; ask Grant Grundler.
> 
> > Anyway, looks like serial consoles will always work. So at least this can
> be
> > done for kdump case (CONFIG_CRASH_DUMP) and not generic kernel. Or, as I
> > mentioned in previous mail, while pre-loading capture kernel, pass a
> command
> > line parameter containing pci dev id of console and capture kernel does not
> 
> > disable interrupts on this console.  
> 
> I suspect you're right that implementing this only in kdump kernels will 
> work okay.
> 
> For people interesting in reading some old threads on the subject, here 
> are some pointers:
> 
> http://marc.theaimsgroup.com/?l=linux-usb-devel&m=111055702309788&w=2
> 
> http://marc.theaimsgroup.com/?l=linux-kernel&m=98383052711171&w=2

I browsed through the discussion threads quickly. Previous proposal included
disabling the DMA as well from the devices. Currently, for kdump, we are not 
looking at disabling the DMA from the devices. So far have not run into 
any problems due to ongoing DMA (Need to look into IOMMU reprogramming aspect
though). Following is a snippet from one of the discussion threads.


|List:       linux-usb-devel
|Subject:    [linux-usb-devel] Re: PCI device initialization and USB
early-handoff
|From:       Grant Grundler <grundler () parisc-linux ! org>
|Date:       2005-03-11 18:12:57
|Message-ID: <20050311181257.GB15070 () colo ! lackof ! org>
|[Download message RAW]
.....

|> > Is it feasible to have the PCI device initialization sequence disable DMA
|> > and IRQs from the device?  This could solve the problems we've been seeing
|> > with non-quiescent devices sharing an IRQ line at startup.

|two potential issues here:
|o ISTR VGA devices may not like disabling Bus Master bit in the command reg.
| But I'm blissfully ignorant of all the issues around VGA and someone
| else will have to comment on that.
|
|o platform devices (e.g. bridges) that don't have PCI drivers to re-enable
|  them later. "transperent" Bridges are the only example I can come up with
|  now but expect more to come out of the woodwork as this gets widely
|  tested. Trolling through PCI quirks might flag some of the known ones.
|  I would expect a few more to show up with this change.

|hth,
|grant


o Bus Master bit is not being disabled, only interrupt generation will be 
  disabled and looks like at least VGA and serial consoles are not impacted 
  due to disabling of interrupt. Any other consoles which require interrupt
  to be enabled for their working????

o As per pci-to-pci bridge architecture specification revision 1.2, 
  interrupt disable bit in PCI-PCI bridge is optional and if implemented,
  it will disable interrupt generation from bridge but will have no effect
  on interrupts that the bridge forwards from the PCI devices on the 
  secondary bus.

  So even if interrupts are disabled on PCI-PCI bridge, interrupts generated
  by PCI devices on secondary bus are not blocked and I hope device should
  be working fine.


The whole idea is that currently this change is kdump specific. Ofcourse there
shall be issues which are not known yet and more devices might not
work for kdump kernels. But at the same time kdump kernels are not supposed to
do a great deal except capture and save the dump. So this change might not
be of a big concern even if some devices don't work as long as kdump kernel
can boot. 

Disabling interrupts at PCI level should increase the reliability of capturing
the dump on newer machines with hardware compliant with PCI 2.3 or higher. 
Booting a kdump kernel with reduced functionality should always be better than 
not booting at all. 

Thanks
Vivek

Re: [RFC/PATCH] Kdump: Disabling PCI interrupts in capture kernel

[Grant Grundler]

On Mon, Jun 06, 2005 at 11:07:17PM -0400, Vivek Goyal wrote:
>   So even if interrupts are disabled on PCI-PCI bridge, interrupts generated
>   by PCI devices on secondary bus are not blocked and I hope device should
>   be working fine.

How did you plan on disabling interrupts?
Did you see the MSI discussion that going on now in linux-pci mailing list?

> But at the same time kdump kernels are not supposed to
> do a great deal except capture and save the dump.

I'd think you want to stop DMA for all devices.
Just to prevent them from messing more with memory
that you want to dump - ie get a consistent snapshot.
Leaving VGA devices alone should be safe.

> Disabling interrupts at PCI level should increase the reliability of capturing
> the dump on newer machines with hardware compliant with PCI 2.3 or higher. 

*lots* of PCI devices predate PCI2.3. Possibly even the majority.

hth,
grant

Re: [RFC/PATCH] Kdump: Disabling PCI interrupts in capture kernel

[Eric W. Biederman]

Grant Grundler <grundler@parisc-linux.org> writes:

> On Mon, Jun 06, 2005 at 11:07:17PM -0400, Vivek Goyal wrote:
> >   So even if interrupts are disabled on PCI-PCI bridge, interrupts generated
> >   by PCI devices on secondary bus are not blocked and I hope device should
> >   be working fine.
> 
> How did you plan on disabling interrupts?
> Did you see the MSI discussion that going on now in linux-pci mailing list?
> 
> > But at the same time kdump kernels are not supposed to
> > do a great deal except capture and save the dump.
> 
> I'd think you want to stop DMA for all devices.
> Just to prevent them from messing more with memory
> that you want to dump - ie get a consistent snapshot.
> Leaving VGA devices alone should be safe.
> 
> > Disabling interrupts at PCI level should increase the reliability of capturing
> 
> > the dump on newer machines with hardware compliant with PCI 2.3 or higher. 
> 
> *lots* of PCI devices predate PCI2.3. Possibly even the majority.

In general generic hardware bits for disabling DMA, disabling interrupts
and the like are all advisory.  With the current architecture things
will work properly even if you don't manage to disable DMA (assuming
you don't reassign IOMMU entries at least).

Non-shared interrupts are not a problem as they can fairly safely
be disabled at the interrupt controller.

Shared interrupts are an interesting case.  The simplest solution I can
think of for a crash dump capture kernel is to periodically poll
the hardware, as if all interrupts are shared.  At that level
I think we could get away with ignoring all hardware interrupt sources.

Does anyone know of a anything that would break by always polling
the hardware?  I guess there could be a problem with drivers
that don't understand shared interrupts, are there enough of those
to be an issue.

Eric

Re: [RFC/PATCH] Kdump: Disabling PCI interrupts in capture kernel

[Grant Grundler]

On Tue, Jun 07, 2005 at 03:59:18AM -0600, Eric W. Biederman wrote:
> > *lots* of PCI devices predate PCI2.3. Possibly even the majority.
> 
> In general generic hardware bits for disabling DMA, disabling interrupts
> and the like are all advisory.  With the current architecture things
> will work properly even if you don't manage to disable DMA (assuming
> you don't reassign IOMMU entries at least).

ISTR, pSeries (IBM), some alpha, some sparc64, and parisc (64-bit) require
use of the IOMMU for *any* DMA. ie IOMMU entries need to be programmed.
Probably want to make a choice to ignore those arches for now
or sort out how to deal with an IOMMU.

> Shared interrupts are an interesting case.  The simplest solution I can
> think of for a crash dump capture kernel is to periodically poll
> the hardware, as if all interrupts are shared.  At that level
> I think we could get away with ignoring all hardware interrupt sources.

Yes, that's perfectly ok. We are no longer in a multitasking env.

> Does anyone know of a anything that would break by always polling
> the hardware?  I guess there could be a problem with drivers
> that don't understand shared interrupts, are there enough of those
> to be an issue.

PCI requires drivers support Shared IRQs.
A few oddballs might be broken but I expect networking/mass storage
drivers get this right.

grant

Re: [Fastboot] Re: [RFC/PATCH] Kdump: Disabling PCI interrupts in capture kernel

[Eric W. Biederman]

Grant Grundler <grundler@parisc-linux.org> writes:

> On Tue, Jun 07, 2005 at 03:59:18AM -0600, Eric W. Biederman wrote:
> > > *lots* of PCI devices predate PCI2.3. Possibly even the majority.
> > 
> > In general generic hardware bits for disabling DMA, disabling interrupts
> > and the like are all advisory.  With the current architecture things
> > will work properly even if you don't manage to disable DMA (assuming
> > you don't reassign IOMMU entries at least).
> 
> ISTR, pSeries (IBM), some alpha, some sparc64, and parisc (64-bit) require
> use of the IOMMU for *any* DMA. ie IOMMU entries need to be programmed.
> Probably want to make a choice to ignore those arches for now
> or sort out how to deal with an IOMMU.

The howto deal with an IOMMU has been sorted out but so far no one 
has actually done it.  What has been discussed previously is simply
reserving a handful of IOMMU entries, and then only using those
in the crash recover kernel.  This is essentially what we do with DMA
on architectures that don't have an IOMMU and it seems quite safe
enough there.

> > Shared interrupts are an interesting case.  The simplest solution I can
> > think of for a crash dump capture kernel is to periodically poll
> > the hardware, as if all interrupts are shared.  At that level
> > I think we could get away with ignoring all hardware interrupt sources.
> 
> Yes, that's perfectly ok. We are no longer in a multitasking env.

Well we are at least capable of multitasking but that is no longer the
primary focus.  Having polling as at least an option should make
debugging easier.  Last I looked Andrews kernel hand an irqpoll option
to do something very like this.

> > Does anyone know of a anything that would break by always polling
> > the hardware?  I guess there could be a problem with drivers
> > that don't understand shared interrupts, are there enough of those
> > to be an issue.
> 
> PCI requires drivers support Shared IRQs.
> A few oddballs might be broken but I expect networking/mass storage
> drivers get this right.

Agreed.  Which means any drivers we really need for dumping the system
should be fine.  If the drivers don't work in that mode at least the
concept of fixing it won't be controversial.


Eric

Re: [Fastboot] Re: [RFC/PATCH] Kdump: Disabling PCI interrupts in capture kernel

[Grant Grundler]

On Tue, Jun 07, 2005 at 12:42:42PM -0600, Eric W. Biederman wrote:
> The howto deal with an IOMMU has been sorted out but so far no one 
> has actually done it.  What has been discussed previously is simply
> reserving a handful of IOMMU entries,

How? with dma_alloc_consistent() or some special hook?
I'm just curious.

...
>  and then only using those
> in the crash recover kernel.  This is essentially what we do with DMA
> on architectures that don't have an IOMMU and it seems quite safe
> enough there.

Yeah, in general that should be feasible.

One might be able to trivially allocate a small, seperate IO PDIR
just for KDUMP and switch to that. Key thing is it be physically
contiguous in memory. Very little code is involved with IO Pdir
setup for both parisc and IA64. I can't speak for Alpha/sparc/ppc/et al.

...
> Well we are at least capable of multitasking but that is no longer the
> primary focus.  Having polling as at least an option should make
> debugging easier.  Last I looked Andrews kernel hand an irqpoll option
> to do something very like this.

You could run the itimer but I don't see why you should.
Kdump is essentially an embedded linux kernel. It really
doesn't need to be premptive multitasking either.

Anyway, sounds like you guys are on the right track.

thanks,
grant

Re: [Fastboot] Re: [RFC/PATCH] Kdump: Disabling PCI interrupts in capture kernel

[Eric W. Biederman]

Grant Grundler <grundler@parisc-linux.org> writes:

> On Tue, Jun 07, 2005 at 12:42:42PM -0600, Eric W. Biederman wrote:
> > The howto deal with an IOMMU has been sorted out but so far no one 
> > has actually done it.  What has been discussed previously is simply
> > reserving a handful of IOMMU entries,
> 
> How? with dma_alloc_consistent() or some special hook?
> I'm just curious.

We didn't get that far but I believe the idea was a special hook.

> ...
> >  and then only using those
> > in the crash recover kernel.  This is essentially what we do with DMA
> > on architectures that don't have an IOMMU and it seems quite safe
> > enough there.
> 
> Yeah, in general that should be feasible.
> 
> One might be able to trivially allocate a small, seperate IO PDIR
> just for KDUMP and switch to that. Key thing is it be physically
> contiguous in memory. Very little code is involved with IO Pdir
> setup for both parisc and IA64. I can't speak for Alpha/sparc/ppc/et al.

Cool.
 
> ...
> > Well we are at least capable of multitasking but that is no longer the
> > primary focus.  Having polling as at least an option should make
> > debugging easier.  Last I looked Andrews kernel hand an irqpoll option
> > to do something very like this.
> 
> You could run the itimer but I don't see why you should.
> Kdump is essentially an embedded linux kernel. It really
> doesn't need to be premptive multitasking either.

It is mostly a matter of minimizing differences from the norm.

> Anyway, sounds like you guys are on the right track.

Thanks.   It just takes a while for the simple solutions to
get there.

Eric

Re: [Fastboot] Re: [RFC/PATCH] Kdump: Disabling PCI interrupts in capture kernel

[Vivek Goyal]

On Tue, Jun 07, 2005 at 12:42:42PM -0600, Eric W. Biederman wrote:
> Grant Grundler <grundler@parisc-linux.org> writes:
> 
> > On Tue, Jun 07, 2005 at 03:59:18AM -0600, Eric W. Biederman wrote:
> > > > *lots* of PCI devices predate PCI2.3. Possibly even the majority.
> > > 
> > > In general generic hardware bits for disabling DMA, disabling interrupts
> > > and the like are all advisory.  With the current architecture things
> > > will work properly even if you don't manage to disable DMA (assuming
> > > you don't reassign IOMMU entries at least).
> > 
> > ISTR, pSeries (IBM), some alpha, some sparc64, and parisc (64-bit) require
> > use of the IOMMU for *any* DMA. ie IOMMU entries need to be programmed.
> > Probably want to make a choice to ignore those arches for now
> > or sort out how to deal with an IOMMU.
> 
> The howto deal with an IOMMU has been sorted out but so far no one 
> has actually done it.  What has been discussed previously is simply
> reserving a handful of IOMMU entries, and then only using those
> in the crash recover kernel.  This is essentially what we do with DMA
> on architectures that don't have an IOMMU and it seems quite safe
> enough there.
> 
> > > Shared interrupts are an interesting case.  The simplest solution I can
> > > think of for a crash dump capture kernel is to periodically poll
> > > the hardware, as if all interrupts are shared.  At that level
> > > I think we could get away with ignoring all hardware interrupt sources.
> > 
> > Yes, that's perfectly ok. We are no longer in a multitasking env.
> 
> Well we are at least capable of multitasking but that is no longer the
> primary focus.  Having polling as at least an option should make
> debugging easier.  Last I looked Andrews kernel hand an irqpoll option
> to do something very like this.
> 

If I understand this right, the idea is that let all irqs be masked (except
timer one) and invoke all the irq handlers whenever a timer interrupt occurs.
This will automatcally be equivalent to drivers polling their devices for
any interrupt.

As you mentioned that irqpoll option comes close. If enabled, it invokes
all the irq handlers on every timer interrupt (IRQ0). The only difference is 
that irqs are not masked (until and unless kernel masks these due to excessive
unhandled interrupts). 

I tried booting kdump kernel with irqpoll option. It seems to be going 
little bit ahead than previous point of failure (boot without irqpoll) but
panics later. Following is the stack trace.

mptbase: Initiating ioc0 bringup
Unable to handle kernel NULL pointer dereference at virtual address 00000608
 printing eip:
c11e1b73
*pde = 00000000
Oops: 0000 [#1]
Modules linked in:
CPU:    0
EIP:    0060:[<c11e1b73>]    Not tainted VLI
EFLAGS: 00010006   (2.6.12-rc6-mm1-16M)
EIP is at mptscsih_io_done+0x23/0x350
eax: c1778400   ebx: 00000000   ecx: 00000600   edx: 00000250
esi: 00000000   edi: 0000000e   ebp: 00000001   esp: c15efcbc
ds: 007b   es: 007b   ss: 0068
Process swapper (pid: 1, threadinfo=c15ee000 task=c147fa00)
Stack: c15f02d1 c1116cfd c1116d60 c15df660 00000000 0000006c 00000250 00000000
       00000000 0000000e 00000001 c11db9ea c1778400 00000600 00000000 00000000
       00000600 c1788aa0 c1410e80 00000009 00000001 c1039e7c 00000009 c1778400
Call Trace:
 [<c1116cfd>] acpi_ev_gpe_detect+0x83/0x10f
 [<c1116d60>] acpi_ev_gpe_detect+0xe6/0x10f
 [<c11db9ea>] mpt_interrupt+0xfa/0x1e0
 [<c1039e7c>] misrouted_irq+0xec/0x100
 [<c103a007>] note_interrupt+0xb7/0xf0
 [<c10399c4>] __do_IRQ+0xe4/0xf0
 [<c1004e09>] do_IRQ+0x19/0x30
 [<c1003246>] common_interrupt+0x1a/0x20
 [<c1018d8f>] release_console_sem+0x3f/0xa0
 [<c1018c27>] vprintk+0x177/0x220
 [<c126ecbd>] pci_read+0x3d/0x50
 [<c1101a0a>] kobject_get+0x1a/0x30
 [<c116b676>] get_device+0x16/0x30
 [<c110b95a>] pci_dev_get+0x1a/0x30
 [<c1018aa7>] printk+0x17/0x20
 [<c11dcbeb>] mpt_do_ioc_recovery+0x4b/0x540
 [<c11dc4ff>] mpt_attach+0x2ef/0x690
 [<c11e6e03>] mptspi_probe+0x23/0x3e0
 [<c110b4f2>] pci_device_probe_static+0x52/0x70
 [<c110b54c>] __pci_device_probe+0x3c/0x50
 [<c110b58f>] pci_device_probe+0x2f/0x50
 [<c116cb68>] driver_probe_device+0x38/0xb0
 [<c116cc60>] __driver_attach+0x0/0x50
 [<c116cca7>] __driver_attach+0x47/0x50
 [<c116c229>] bus_for_each_dev+0x69/0x80
 [<c116ccd6>] driver_attach+0x26/0x30
 [<c116cc60>] __driver_attach+0x0/0x50
 [<c116c6d8>] bus_add_driver+0x88/0xc0
 [<c110b6d0>] pci_device_shutdown+0x0/0x30
 [<c110b857>] pci_register_driver+0x67/0x90
 [<c142a890>] mptspi_init+0xa0/0xb0
 [<c11e37a0>] mptscsih_ioc_reset+0x0/0x170
 [<c141483c>] do_initcalls+0x2c/0xc0
 [<c142d5fa>] sock_init+0x2a/0x40
 [<c1000290>] init+0x0/0x100
 [<c10002b5>] init+0x25/0x100
 [<c10013a0>] kernel_thread_helper+0x0/0x10
 [<c10013a5>] kernel_thread_helper+0x5/0x10
Code: 00 8d bc 27 00 00 00 00 55 57 56 53 83 ec 1c 8b 44 24 30 8b 4c 24 34 8b 7
 <0>Kernel panic - not syncing: Fatal exception in interrupt


Thanks
Vivek

Re: [Fastboot] Re: [RFC/PATCH] Kdump: Disabling PCI interrupts in capture kernel

[Vivek Goyal]

> > 
> > > > Shared interrupts are an interesting case.  The simplest solution I can
> > > > think of for a crash dump capture kernel is to periodically poll
> > > > the hardware, as if all interrupts are shared.  At that level
> > > > I think we could get away with ignoring all hardware interrupt sources.
> > > 
> > > Yes, that's perfectly ok. We are no longer in a multitasking env.
> > 
> > Well we are at least capable of multitasking but that is no longer the
> > primary focus.  Having polling as at least an option should make
> > debugging easier.  Last I looked Andrews kernel hand an irqpoll option
> > to do something very like this.
> > 
> 
> If I understand this right, the idea is that let all irqs be masked (except
> timer one) and invoke all the irq handlers whenever a timer interrupt occurs.
> This will automatcally be equivalent to drivers polling their devices for
> any interrupt.
> 
> As you mentioned that irqpoll option comes close. If enabled, it invokes
> all the irq handlers on every timer interrupt (IRQ0). The only difference is 
> that irqs are not masked (until and unless kernel masks these due to excessive
> unhandled interrupts). 
> 
> I tried booting kdump kernel with irqpoll option. It seems to be going 
> little bit ahead than previous point of failure (boot without irqpoll) but
> panics later. Following is the stack trace.
> 

Second kernel booted fine with MPT_DEBUG_IRQ enabled (with irqpoll option). 
There were few warning messages though spitted by the code under MPT_DEBUG_IRQ.

Looks like drivers need to be hardened on case to case basis to initialize 
properly even if underlying device is not in a reset state.

Thanks
Vivek