hwo to adjust interrupt?

hwo to adjust interrupt?

[Ming Zhang]

I have this eth1 and aic7xxx share 1 interrupt thus have impact on
performance, anyway i can change it? thx!


[root@sc420 root]# cat /proc/interrupts
           CPU0       CPU1
  0:     378525     402791    IO-APIC-edge  timer
  1:         38         26    IO-APIC-edge  i8042
  4:          7          0    IO-APIC-edge  serial
  8:          3          2    IO-APIC-edge  rtc
  9:          0          0   IO-APIC-level  acpi
 14:       2325        932    IO-APIC-edge  ide0
 16:    1330261          0   IO-APIC-level  eth1, aic7xxx
 17:       2917          8   IO-APIC-level  aic7xxx
 18:          3          0   IO-APIC-level  uhci_hcd:usb4
 20:       7577       6597   IO-APIC-level  libata
 21:          0          0   IO-APIC-level  ehci_hcd:usb1, uhci_hcd:usb2
 22:          0          0   IO-APIC-level  uhci_hcd:usb3
 23:          0          0   IO-APIC-level  uhci_hcd:usb5
NMI:    2017738    1165555
LOC:     781237     781288
ERR:          0
MIS:          0

Re: hwo to adjust interrupt?

[Arjan van de Ven]

On Thu, 2005-08-11 at 18:31 -0400, Ming Zhang wrote:
> I have this eth1 and aic7xxx share 1 interrupt thus have impact on
> performance, anyway i can change it? thx!

realistically, either you need to find a bios setting tweak, or you need
to move one of the two to a different pci slot in the machine.


(Not that it's a big problem, sharing irqs on a pci system isn't too big
a deal)

Re: hwo to adjust interrupt?

[Ming Zhang]

On Fri, 2005-08-12 at 09:14 +0200, Arjan van de Ven wrote:
> On Thu, 2005-08-11 at 18:31 -0400, Ming Zhang wrote:
> > I have this eth1 and aic7xxx share 1 interrupt thus have impact on
> > performance, anyway i can change it? thx!
> 
> realistically, either you need to find a bios setting tweak, or you need
> to move one of the two to a different pci slot in the machine.
> 

i could not find any bios setting in the bios. it is a dell sc420. i
will try to move card around and see what happen. thx!


> 
> (Not that it's a big problem, sharing irqs on a pci system isn't too big
> a deal)
> 

i use it as my iscsi testbed so the sole activities are disk&network.
that is why i am worrying about it.

do you think it is ok to share? i think at least kernel need to find 1
irq handler to process the handler, then it have 1/2 chance (roughly
assume 1 network interrupt corresponding to 1 storage io interrupt, of
course this is not true in reality) to find that this is not the right
one and then reload another handler? i guess this will be critical when
# is huge like this. (i ran might 10 minutes with iometer on an iscsi
target.)

16:    1330261          0   IO-APIC-level  eth1, aic7xxx



Ming

Re: hwo to adjust interrupt?

[Bryan Henderson]

>(Not that it's a big problem, sharing irqs on a pci system isn't too big
>a deal)

Just how big a deal is it?  How much extra work does Linux have to do for 
each interrupt?

Re: hwo to adjust interrupt?

[Arjan van de Ven]

On Fri, 2005-08-12 at 09:59 -0700, Bryan Henderson wrote:
> >(Not that it's a big problem, sharing irqs on a pci system isn't too big
> >a deal)
> 
> Just how big a deal is it?  How much extra work does Linux have to do for 
> each interrupt?

usually 1 pci mmio read; the rest is negliable.

Re: hwo to adjust interrupt?

[Bryan Henderson]

>>How much extra work does Linux have to do for each interrupt?
>
>usually 1 pci mmio read; the rest is negligible.

I was hoping you would cater better to my ignorance of how PCI interrupt 
handling works in Linux.

Is it the case that Linux invokes the registered interrupt handler of each 
of the drivers for the devices that share the interrupt, and each does an 
mmio read of its device to find out if it had reason to generate an 
interrupt?  So the waste is that extra call, and you're saying the CPU 
instructions involved are negligible compared to the mmio read?

Are these level-sensitive interrupts, so that if both devices need service 
at the same time, they generate just one interrupt and neither device 
driver call is wasted?

Re: hwo to adjust interrupt?

[Ming Zhang]

On Fri, 2005-08-12 at 10:35 -0700, Bryan Henderson wrote:
> >>How much extra work does Linux have to do for each interrupt?
> >
> >usually 1 pci mmio read; the rest is negligible.
> 
> I was hoping you would cater better to my ignorance of how PCI interrupt 
> handling works in Linux.
> 
> Is it the case that Linux invokes the registered interrupt handler of each 
> of the drivers for the devices that share the interrupt, and each does an 
> mmio read of its device to find out if it had reason to generate an 
> interrupt?  So the waste is that extra call, and you're saying the CPU 
> instructions involved are negligible compared to the mmio read?
> 
> Are these level-sensitive interrupts, so that if both devices need service 
> at the same time, they generate just one interrupt and neither device 
> driver call is wasted?
> 

i guess so. 

also sound like the intr overhead is not that big as i assumed.
 
so if i really want to make myself comfortable, i can disable the share
interrupt when register interrupt in driver code, rite?

ming

Re: hwo to adjust interrupt?

[Arjan van de Ven]

On Fri, 2005-08-12 at 10:35 -0700, Bryan Henderson wrote:
> >>How much extra work does Linux have to do for each interrupt?
> >
> >usually 1 pci mmio read; the rest is negligible.
> 
> I was hoping you would cater better to my ignorance of how PCI interrupt 
> handling works in Linux.
> 
> Is it the case that Linux invokes the registered interrupt handler of each 
> of the drivers for the devices that share the interrupt, and each does an 
> mmio read of its device to find out if it had reason to generate an 
> interrupt?  So the waste is that extra call, and you're saying the CPU 
> instructions involved are negligible compared to the mmio read?

yes. A function call is like half a cycle (a function pointer call is
maybe 40) an mmio read is a lot more

> Are these level-sensitive interrupts, so that if both devices need service 
> at the same time, they generate just one interrupt and neither device 
> driver call is wasted?

ok this is more complex, but if 2 cards raise it quickly after
eachother, before the ISR has run, then you only get the handler called
once afaik.

Re: hwo to adjust interrupt?

[Ming Zhang]

On Fri, 2005-08-12 at 19:51 +0200, Arjan van de Ven wrote:
> On Fri, 2005-08-12 at 10:35 -0700, Bryan Henderson wrote:
> > >>How much extra work does Linux have to do for each interrupt?
> > >
> > >usually 1 pci mmio read; the rest is negligible.
> > 
> > I was hoping you would cater better to my ignorance of how PCI interrupt 
> > handling works in Linux.
> > 
> > Is it the case that Linux invokes the registered interrupt handler of each 
> > of the drivers for the devices that share the interrupt, and each does an 
> > mmio read of its device to find out if it had reason to generate an 
> > interrupt?  So the waste is that extra call, and you're saying the CPU 
> > instructions involved are negligible compared to the mmio read?
> 
> yes. A function call is like half a cycle (a function pointer call is
> maybe 40) an mmio read is a lot more
 a bit off topic, but where u get these information? thx


> 
> > Are these level-sensitive interrupts, so that if both devices need service 
> > at the same time, they generate just one interrupt and neither device 
> > driver call is wasted?
> 
> ok this is more complex, but if 2 cards raise it quickly after
> eachother, before the ISR has run, then you only get the handler called
> once afaik.
so it is possible that both card raise intr quicker than isr run, then
both isr will do mmio and go ahead, but then what is

device 1 raise intr
device 2 isr run and not belong to it, so quit, but before it clear intr
and quit
device 2 raise intr
device 1 isr run and handle device 1, then clear intr and quit.

then device 2 intr lost?

or i understand this wrong. :)

ming


> 
> 
> 

Re: hwo to adjust interrupt?

[Bryan Henderson]

>so it is possible that both card raise intr quicker than isr run, then
>both isr will do mmio and go ahead, but then what is
>
>device 1 raise intr
>device 2 isr run and not belong to it, so quit, but before it clear intr
>and quit
>device 2 raise intr
>device 1 isr run and handle device 1, then clear intr and quit.
>
>then device 2 intr lost?

As these are level sensitive interrupt requests, they can't be lost.  As 
long as Device 2 is requesting service,
the shared IRQ is raised and at some point Linux will call Device 2's 
interrupt handler.  Either the first level IRQ handler will keep calling 
all the interrupt handlers that service that IRQ until the IRQ subsides, 
or it will unmask the IRQ, causing an immediate new interrupt and the 
first level interrupt handler runs again and calls all the device drivers 
again.

The raised IRQ says, "Device 1 or Device 2 needs service."  The device 
drivers clear the service request of the individual device; as long as 
either device requests service, the IRQ is raised.

--
Bryan Henderson                     IBM Almaden Research Center
San Jose CA                         Filesystems

Re: hwo to adjust interrupt?

[Ming Zhang]

ic. thanks a lot for explanation.

so these are level sensitive interrupt.

i need to find a hw book now...

Ming

On Mon, 2005-08-15 at 11:47 -0700, Bryan Henderson wrote:
> >so it is possible that both card raise intr quicker than isr run, then
> >both isr will do mmio and go ahead, but then what is
> >
> >device 1 raise intr
> >device 2 isr run and not belong to it, so quit, but before it clear intr
> >and quit
> >device 2 raise intr
> >device 1 isr run and handle device 1, then clear intr and quit.
> >
> >then device 2 intr lost?
> 
> As these are level sensitive interrupt requests, they can't be lost.  As 
> long as Device 2 is requesting service,
> the shared IRQ is raised and at some point Linux will call Device 2's 
> interrupt handler.  Either the first level IRQ handler will keep calling 
> all the interrupt handlers that service that IRQ until the IRQ subsides, 
> or it will unmask the IRQ, causing an immediate new interrupt and the 
> first level interrupt handler runs again and calls all the device drivers 
> again.
> 
> The raised IRQ says, "Device 1 or Device 2 needs service."  The device 
> drivers clear the service request of the individual device; as long as 
> either device requests service, the IRQ is raised.
> 
> --
> Bryan Henderson                     IBM Almaden Research Center
> San Jose CA                         Filesystems
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: hwo to adjust interrupt?

[Ming Zhang]

i tried to remove the SA_SHIRQ option when request_irq(), unfortunately
it seems that the IO-APIC will only try to allocate same # to it and
once that slot is used, the request_irq will fail.

ming


On Fri, 2005-08-12 at 10:35 -0700, Bryan Henderson wrote:
> >>How much extra work does Linux have to do for each interrupt?
> >
> >usually 1 pci mmio read; the rest is negligible.
> 
> I was hoping you would cater better to my ignorance of how PCI interrupt 
> handling works in Linux.
> 
> Is it the case that Linux invokes the registered interrupt handler of each 
> of the drivers for the devices that share the interrupt, and each does an 
> mmio read of its device to find out if it had reason to generate an 
> interrupt?  So the waste is that extra call, and you're saying the CPU 
> instructions involved are negligible compared to the mmio read?
> 
> Are these level-sensitive interrupts, so that if both devices need service 
> at the same time, they generate just one interrupt and neither device 
> driver call is wasted?
> 

Re: hwo to adjust interrupt?

[Matthew Wilcox]

On Fri, Aug 12, 2005 at 03:05:51PM -0400, Ming Zhang wrote:
> i tried to remove the SA_SHIRQ option when request_irq(), unfortunately
> it seems that the IO-APIC will only try to allocate same # to it and
> once that slot is used, the request_irq will fail.

Yes.  These two interrupts are physically on the same wire attached to
the IO-APIC.  You need to put the card in a different slot to change
which interrupt it gets.

-- 
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain

Re: hwo to adjust interrupt?

[Ming Zhang]

On Fri, 2005-08-12 at 20:14 +0100, Matthew Wilcox wrote:
> On Fri, Aug 12, 2005 at 03:05:51PM -0400, Ming Zhang wrote:
> > i tried to remove the SA_SHIRQ option when request_irq(), unfortunately
> > it seems that the IO-APIC will only try to allocate same # to it and
> > once that slot is used, the request_irq will fail.
> 
> Yes.  These two interrupts are physically on the same wire attached to
> the IO-APIC.  You need to put the card in a different slot to change
> which interrupt it gets.
> 
thanks.

i tried to move aic7xxx into another slot and now it uses 17 and 18
instead of previously 16 17.

thus my question is how many intr available in APIC? what if i have a
quad port nic card and 1-2 scsi dual port controllers on board? then
they must share anyway?

[root@sc420 root]# cat /proc/interrupts
           CPU0       CPU1
  0:     103368      60018    IO-APIC-edge  timer
  1:         80         68    IO-APIC-edge  i8042
  4:          7          0    IO-APIC-edge  serial
  8:          3          0    IO-APIC-edge  rtc
  9:          0          0   IO-APIC-level  acpi
 14:        494          0    IO-APIC-edge  ide0
 16:        740          0   IO-APIC-level  eth1
 17:         15          0   IO-APIC-level  aic7xxx
 18:         28         43   IO-APIC-level  uhci_hcd:usb4, eth0, aic7xxx
 20:       4178        179   IO-APIC-level  libata
 21:          0          0   IO-APIC-level  ehci_hcd:usb1, uhci_hcd:usb2
 22:          0          0   IO-APIC-level  uhci_hcd:usb3
 23:          0          0   IO-APIC-level  uhci_hcd:usb5
NMI:          0          0
LOC:     163279     163330
ERR:          0
MIS:          0