Re: What determines which interrupts are shared under Linux?

Re: What determines which interrupts are shared under Linux?

[Robert Hancock]

Roger Heflin wrote:
> Hello,
> 
> On Linux when interrupts are defined similar to below, what defines say
> ide2, ide3 to be on the same interrupt?    The bios, linux, the driver 
> using
> the interrupt?    And can that be controlled/overrode at the 
> kernel/driver level?
> 

Typically this is determined by the hardware routing of the interrupt 
lines on the motherboard, occasionally the BIOS. There isn't usually 
much that can be done about the sharing at the kernel level.

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

What determines which interrupts are shared under Linux?

[Roger Heflin]

Hello,

On Linux when interrupts are defined similar to below, what defines say
ide2, ide3 to be on the same interrupt?    The bios, linux, the driver using
the interrupt?    And can that be controlled/overrode at the 
kernel/driver level?

I have identified that the disks that are shared on ide2, ide3 do funny
things when both are being heavily used (dma_expiry), this is an older 
driver versions
but I have experienced it before with a lot newer driver, and a bios 
adjustment
previously fixed a similar issue, so that may be what is needed in this 
case also,
I am not sure how they fixed it, but I suspect that the setup the interrupt
to not be shared.   I have a large number of machines and under heavy 
loads all
seem to duplicate the issue, and it always happens with the disks on 
ide2/ide3,
never on the disk connected to ide4.

            CPU0       CPU1       CPU2       CPU3
   0:   56616921    5359998    7002142     938817          XT-PIC  timer
   1:          8         88         96          0    IO-APIC-edge  i8042
   2:          0          0          0          0          XT-PIC  cascade
   4:       2091        100        208       2477    IO-APIC-edge  serial
   8:          0          0          0          0    IO-APIC-edge  rtc
   9:          0          0          0          0   IO-APIC-level  acpi
  20:          0          0          0          0   IO-APIC-level  ehci_hcd
  21:          0        950     401419     414482   IO-APIC-level  ide4, 
ohci_hcd
  22:       1165    1704243     576247       6796   IO-APIC-level  ide2, 
ide3
  47:      65971          0          0          0   IO-APIC-level  eth0
NMI:          1          1          1          1
LOC:   69904264   69877733   69879541   69901903
ERR:          0
MIS:        105


                                       Roger

Re: What determines which interrupts are shared under Linux?

[Sergei Shtylyov]

Hello,

Roger Heflin wrote:

> On Linux when interrupts are defined similar to below, what defines say
> ide2, ide3 to be on the same interrupt? The bios, linux, the driver 
> using the interrupt?

    In this particular case (I suppoer both belong to the same IDE conrtoller) 
the hardware design determines this in part and software in another part.
If the PCI IDE controller was in the leagcy mode (it should be ide0/1 then), 
the interrupts would've been 14 and 15. But in the native PCI mode, both 
chanels share the same interrupt level (if they're both in native mode that is).

> And can that be controlled/overrode at the kernel/driver level?

    The only possibility might be do disable ide0/1 in the BIOS I guess.

> I have identified that the disks that are shared on ide2, ide3 do funny
> things when both are being heavily used (dma_expiry), this is an older 
> driver versions
> but I have experienced it before with a lot newer driver, and a bios 
> adjustment
> previously fixed a similar issue, so that may be what is needed in this 
> case also,
> I am not sure how they fixed it, but I suspect that the setup the interrupt
> to not be shared.

    I doubt that your suspicions are justified.

>   I have a large number of machines and under heavy 
> loads all
> seem to duplicate the issue, and it always happens with the disks on 
> ide2/ide3,
> never on the disk connected to ide4.

    BTW, you never named your particular IDE hardware.

>            CPU0       CPU1       CPU2       CPU3
>   0:   56616921    5359998    7002142     938817          XT-PIC  timer
>   1:          8         88         96          0    IO-APIC-edge  i8042
>   2:          0          0          0          0          XT-PIC  cascade
>   4:       2091        100        208       2477    IO-APIC-edge  serial
>   8:          0          0          0          0    IO-APIC-edge  rtc
>   9:          0          0          0          0   IO-APIC-level  acpi
>  20:          0          0          0          0   IO-APIC-level  ehci_hcd
>  21:          0        950     401419     414482   IO-APIC-level  ide4, 
> ohci_hcd
>  22:       1165    1704243     576247       6796   IO-APIC-level  ide2, 
> ide3
>  47:      65971          0          0          0   IO-APIC-level  eth0
> NMI:          1          1          1          1
> LOC:   69904264   69877733   69879541   69901903
> ERR:          0
> MIS:        105

MBR, Sergei

Re: What determines which interrupts are shared under Linux?

[Roger Heflin]

Sergei Shtylyov wrote:
> Hello,
> 
>> I have identified that the disks that are shared on ide2, ide3 do funny
>> things when both are being heavily used (dma_expiry), this is an older 
>> driver versions
>> but I have experienced it before with a lot newer driver, and a bios 
>> adjustment
>> previously fixed a similar issue, so that may be what is needed in 
>> this case also,
>> I am not sure how they fixed it, but I suspect that the setup the 
>> interrupt
>> to not be shared.
> 
>    I doubt that your suspicions are justified.
> 
>>   I have a large number of machines and under heavy loads all
>> seem to duplicate the issue, and it always happens with the disks on 
>> ide2/ide3,
>> never on the disk connected to ide4.
> 
>    BTW, you never named your particular IDE hardware.

NVIDIA, CK804 chipset.

And the disks are on the sata part, and are found as e g and i so if
they are in legacy mode, it would make me suspect that the disks believe
they are on separate channels, and the performance characteristics appear
to confirm that as the disks don't seem to interfere with each other.

> 
>>            CPU0       CPU1       CPU2       CPU3
>>   0:   56616921    5359998    7002142     938817          XT-PIC  timer
>>   1:          8         88         96          0    IO-APIC-edge  i8042
>>   2:          0          0          0          0          XT-PIC  cascade
>>   4:       2091        100        208       2477    IO-APIC-edge  serial
>>   8:          0          0          0          0    IO-APIC-edge  rtc
>>   9:          0          0          0          0   IO-APIC-level  acpi
>>  20:          0          0          0          0   IO-APIC-level  
>> ehci_hcd
>>  21:          0        950     401419     414482   IO-APIC-level  
>> ide4, ohci_hcd
>>  22:       1165    1704243     576247       6796   IO-APIC-level  
>> ide2, ide3
>>  47:      65971          0          0          0   IO-APIC-level  eth0
>> NMI:          1          1          1          1
>> LOC:   69904264   69877733   69879541   69901903
>> ERR:          0
>> MIS:        105
> 
> MBR, Sergei
> 

Re: What determines which interrupts are shared under Linux?

[Len Brown]

On Tuesday 15 August 2006 10:17, Roger Heflin wrote:
>  22:       1165    1704243     576247       6796   IO-APIC-level  ide2,  ide3

The first thing that determines interrupt sharing is where the physical wires from
the devices are routed.  If these ide controllers are add-on boards, then you
could try moving them between PCI slots -- as the slots generally have different
primary interrupt lines.  When the board runs out of unique lines,
they are generally wired to re-use lines on different slots.
The manual for the board will generally tell you how the lines
are routed.

If the devices are using the same physical interrupt line, then it
is not possible for software or BIOS to move them to different
lines.

If these devices are functions in the same device, it is possible
that there is some internal (BIOS or firmware) setting to tell it to
use two interrupt wires instead of one.

Once the wire for the device is determined, it is up to the BIOS
and the OS to program (or not) an interrupt router to assign
the wire to an interrupt input pin.  Your dmesg will tell us
if this is happening on this board.

-Len

Re: What determines which interrupts are shared under Linux?

[Alan Cox]

Ar Maw, 2006-08-15 am 09:17 -0500, ysgrifennodd Roger Heflin:
> On Linux when interrupts are defined similar to below, what defines say
> ide2, ide3 to be on the same interrupt?    The bios, linux, the driver using
> the interrupt?    And can that be controlled/overrode at the 
> kernel/driver level?

Only with a soldering iron. They are the way the system is wired. Moving
boards between slots may change the IRQ allocation.

> I have identified that the disks that are shared on ide2, ide3 do funny
> things when both are being heavily used (dma_expiry), this is an older 
> driver versions

That could be occuring just through lack of PCI bus bandwidth.

Re: What determines which interrupts are shared under Linux?

[Roger Heflin]

Alan Cox wrote:
> Ar Maw, 2006-08-15 am 09:17 -0500, ysgrifennodd Roger Heflin:
>> On Linux when interrupts are defined similar to below, what defines say
>> ide2, ide3 to be on the same interrupt?    The bios, linux, the driver using
>> the interrupt?    And can that be controlled/overrode at the 
>> kernel/driver level?
> 
> Only with a soldering iron. They are the way the system is wired. Moving
> boards between slots may change the IRQ allocation.
> 
>> I have identified that the disks that are shared on ide2, ide3 do funny
>> things when both are being heavily used (dma_expiry), this is an older 
>> driver versions
> 
> That could be occuring just through lack of PCI bus bandwidth.
> 
> 
> 

Right now, all controllers are part of the NVIDIA chipset which should
mean the are on the same PCI bandwidth.    The block diagram does indicate
the 4 sata channels as one unit, though I don't know exactly how things
were internally done, performance testing seems to indicate that all
3 I am using are independent hardware as they don't seem to affect each
others load.

I am currently retesting under 2.6.17.8 to see if I have similar issues
there, under that it show interrupts like below, different interrupt 
numbers,
but similar sharing as ata1/ata2, and ata3/ata4 are on the same interrupt.

ata1: SATA max UDMA/133 cmd 0xEC00 ctl 0xE802 bmdma 0xDC00 irq 16
ata2: SATA max UDMA/133 cmd 0xE400 ctl 0xE002 bmdma 0xDC08 irq 16
ata1: SATA link up 3.0 Gbps (SStatus 123)
ata1: dev 0 cfg 49:2f00 82:706b 83:7e61 84:4023 85:7069 86:3e41 87:4023 
88:407f
ata1: dev 0 ATA-7, max UDMA/133, 321672960 sectors: LBA48
ata1: dev 0 configured for UDMA/133
ata2: SATA link up 3.0 Gbps (SStatus 123)
ata2: dev 0 cfg 49:2f00 82:706b 83:7e61 84:4023 85:7069 86:3e41 87:4023 
88:407f
ata2: dev 0 ATA-7, max UDMA/133, 321672960 sectors: LBA48
ata2: dev 0 configured for UDMA/133
ata3: SATA max UDMA/133 cmd 0xD800 ctl 0xD402 bmdma 0xC800 irq 17
ata4: SATA max UDMA/133 cmd 0xD000 ctl 0xCC02 bmdma 0xC808 irq 17
ata3: SATA link up 3.0 Gbps (SStatus 123)
ata3: dev 0 cfg 49:2f00 82:706b 83:7e61 84:4023 85:7069 86:3e41 87:4023 
88:407f
ata3: dev 0 ATA-7, max UDMA/133, 321672960 sectors: LBA48
ata3: dev 0 configured for UDMA/133
ata4: SATA link down (SStatus 0)

                                       Roger

Re: What determines which interrupts are shared under Linux?

[Alan Cox]

Ar Maw, 2006-08-15 am 10:24 -0500, ysgrifennodd Roger Heflin:

> I am currently retesting under 2.6.17.8 to see if I have similar issues
> there, under that it show interrupts like below, different interrupt 
> numbers,
> but similar sharing as ata1/ata2, and ata3/ata4 are on the same interrupt.

Thats what I would expect to see - two channels per PCI device is the
normal layout and they will always share the IRQ.

Re: What determines which interrupts are shared under Linux?

[Roger Heflin]

Alan Cox wrote:
> Ar Maw, 2006-08-15 am 10:24 -0500, ysgrifennodd Roger Heflin:
> 
>> I am currently retesting under 2.6.17.8 to see if I have similar issues
>> there, under that it show interrupts like below, different interrupt 
>> numbers,
>> but similar sharing as ata1/ata2, and ata3/ata4 are on the same interrupt.
> 
> Thats what I would expect to see - two channels per PCI device is the
> normal layout and they will always share the IRQ.
> 
> 

After looking at other information that looks correct.

It looks like the older DMA recovery code never works on this chipset,
once it goes into DMA recovery it never comes out of it.    I am looking
at that to see if anything can be done about it.

The problem I am having is that we cannot use a later kernel because it
has some other issues (MPT fusion driver is 1/4 the speed of the older
one for some unknown reason-this is in 2.6.17, the older MPT fusion works
fine on exactly the same machine/cards/bios-I am not sure of the
actual underlying casue of this slowdown).

On the specific kernel that I have I appear to have both IDE and
sata_nv drivers, is there a way to force things to use sata_nv/libata
rather than the older ide driver for the NVIDIA sata controller?

                                   Roger

Re: What determines which interrupts are shared under Linux?

[Alan Cox]

Ar Maw, 2006-08-15 am 12:47 -0500, ysgrifennodd Roger Heflin:
> On the specific kernel that I have I appear to have both IDE and
> sata_nv drivers, is there a way to force things to use sata_nv/libata
> rather than the older ide driver for the NVIDIA sata controller?

You can build 2.6.18-mm like that (say N to AMD/NVIDIA PATA driver, y to
sata_nv and pata_amd libata drivers). Still have to figure out why your
MPT fusion is slow but if its still slow in 2.6.18-mm then it needs
figuring out anyway

Re: What determines which interrupts are shared under Linux?

[Mark Lord]

Roger Heflin wrote:
>
> It looks like the older DMA recovery code never works on this chipset,
> once it goes into DMA recovery it never comes out of it.    I am looking
> at that to see if anything can be done about it.

You could try booting with ide2=serialize as a kernel parameter.
That should ensure the two channels are never in use simultaneously.
Trades off a bit of performance for a bit of reliability.

Cheers

Re: What determines which interrupts are shared under Linux?

[Alan Cox]

Ar Mer, 2006-08-16 am 18:54 -0400, ysgrifennodd Mark Lord:
> Roger Heflin wrote:
> >
> > It looks like the older DMA recovery code never works on this chipset,
> > once it goes into DMA recovery it never comes out of it. 

DMA recovery is fairly broken in drivers/ide especially if it tries to
change mode. libata does not have this problem and I have no plans to
even try and fix the drivers/ide code for this issue as its a major
piece of work.

Re: What determines which interrupts are shared under Linux?

[Roger Heflin]

Alan Cox wrote:
> Ar Mer, 2006-08-16 am 18:54 -0400, ysgrifennodd Mark Lord:
>> Roger Heflin wrote:
>>> It looks like the older DMA recovery code never works on this chipset,
>>> once it goes into DMA recovery it never comes out of it. 
> 
> DMA recovery is fairly broken in drivers/ide especially if it tries to
> change mode. libata does not have this problem and I have no plans to
> even try and fix the drivers/ide code for this issue as its a major
> piece of work.
> 
> 

If already figured that since I was pretty sure all of it was
being rolled to libata.   I was able to switch that controller and
disk to the libata sata_nv.

                                 Roger

Re: What determines which interrupts are shared under Linux?

[linux-os (Dick Johnson)]

On Tue, 15 Aug 2006, Roger Heflin wrote:

> Hello,
>
> On Linux when interrupts are defined similar to below, what defines say
> ide2, ide3 to be on the same interrupt?    The bios, linux, the driver using
> the interrupt?    And can that be controlled/overrode at the
> kernel/driver level?
>
> I have identified that the disks that are shared on ide2, ide3 do funny
> things when both are being heavily used (dma_expiry), this is an older
> driver versions
> but I have experienced it before with a lot newer driver, and a bios
> adjustment
> previously fixed a similar issue, so that may be what is needed in this
> case also,
> I am not sure how they fixed it, but I suspect that the setup the interrupt
> to not be shared.   I have a large number of machines and under heavy
> loads all
> seem to duplicate the issue, and it always happens with the disks on
> ide2/ide3,
> never on the disk connected to ide4.
>
>            CPU0       CPU1       CPU2       CPU3
>   0:   56616921    5359998    7002142     938817          XT-PIC  timer
>   1:          8         88         96          0    IO-APIC-edge  i8042
>   2:          0          0          0          0          XT-PIC  cascade
>   4:       2091        100        208       2477    IO-APIC-edge  serial
>   8:          0          0          0          0    IO-APIC-edge  rtc
>   9:          0          0          0          0   IO-APIC-level  acpi
>  20:          0          0          0          0   IO-APIC-level  ehci_hcd
>  21:          0        950     401419     414482   IO-APIC-level  ide4,
> ohci_hcd
>  22:       1165    1704243     576247       6796   IO-APIC-level  ide2,
> ide3
>  47:      65971          0          0          0   IO-APIC-level  eth0
> NMI:          1          1          1          1
> LOC:   69904264   69877733   69879541   69901903
> ERR:          0
> MIS:        105
>
>
>                                       Roger

The hardware defines what traces go to whatever chips on the
motherboard and to the feature module sockets. Sometimes,
the ultimate IRQ that these traces go to can be configured in the
BIOS. Sometimes not. The kernel just uses what was set up by
the BIOS, convention, and the PCI devices themselves.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.16.24 on an i686 machine (5592.62 BogoMips).
New book: http://www.AbominableFirebug.com/
_
\x1a\x04

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: What determines which interrupts are shared under Linux?

[Terence Ripperda]

we've seen a lot of problems on ck804 chipsets when multiple devices
share level-triggered interrupts. I think some of the earlier sample
bioses assumed that interrupts would be configured via ACPI, and when
ACPI is not used, the interrupts end up as level-triggered instead of
edge-triggered.

you might investigate whether ACPI is failing to configure your system
(are you on 2.4? updating to 2.6 might help), using the 'noapic'
kernel parameter to force interrupts back to PIC, or the 'irqpoll' kernel
parameter to try to work around the level-triggered interrupts.

Thanks,
Terence


On Tue, Aug 15, 2006 at 09:17:04AM -0500, rheflin@atipa.com wrote:
> Hello,
> 
> On Linux when interrupts are defined similar to below, what defines say
> ide2, ide3 to be on the same interrupt?    The bios, linux, the driver using
> the interrupt?    And can that be controlled/overrode at the 
> kernel/driver level?
> 
> I have identified that the disks that are shared on ide2, ide3 do funny
> things when both are being heavily used (dma_expiry), this is an older 
> driver versions
> but I have experienced it before with a lot newer driver, and a bios 
> adjustment
> previously fixed a similar issue, so that may be what is needed in this 
> case also,
> I am not sure how they fixed it, but I suspect that the setup the interrupt
> to not be shared.   I have a large number of machines and under heavy 
> loads all
> seem to duplicate the issue, and it always happens with the disks on 
> ide2/ide3,
> never on the disk connected to ide4.
> 
>            CPU0       CPU1       CPU2       CPU3
>   0:   56616921    5359998    7002142     938817          XT-PIC  timer
>   1:          8         88         96          0    IO-APIC-edge  i8042
>   2:          0          0          0          0          XT-PIC  cascade
>   4:       2091        100        208       2477    IO-APIC-edge  serial
>   8:          0          0          0          0    IO-APIC-edge  rtc
>   9:          0          0          0          0   IO-APIC-level  acpi
>  20:          0          0          0          0   IO-APIC-level  ehci_hcd
>  21:          0        950     401419     414482   IO-APIC-level  ide4, 
> ohci_hcd
>  22:       1165    1704243     576247       6796   IO-APIC-level  ide2, 
> ide3
>  47:      65971          0          0          0   IO-APIC-level  eth0
> NMI:          1          1          1          1
> LOC:   69904264   69877733   69879541   69901903
> ERR:          0
> MIS:        105
> 
> 
>                                       Roger
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 
> 

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

Re: What determines which interrupts are shared under Linux?

[Sergei Shtylyov]

Hello.

Terence Ripperda wrote:
> we've seen a lot of problems on ck804 chipsets when multiple devices
> share level-triggered interrupts. I think some of the earlier sample
> bioses assumed that interrupts would be configured via ACPI, and when
> ACPI is not used, the interrupts end up as level-triggered instead of
> edge-triggered.

    Edge-triggered *shared* interrupts?! Now that sounds interesting (I'm not 
saying impossible).

WBR, Sergei

Re: What determines which interrupts are shared under Linux?

[Terence Ripperda]

On Tue, Aug 15, 2006 at 09:42:49PM +0400, sshtylyov@ru.mvista.com wrote:
> Hello.
> 
> Terence Ripperda wrote:
> >we've seen a lot of problems on ck804 chipsets when multiple devices
> >share level-triggered interrupts. I think some of the earlier sample
> >bioses assumed that interrupts would be configured via ACPI, and when
> >ACPI is not used, the interrupts end up as level-triggered instead of
> >edge-triggered.
> 
>    Edge-triggered *shared* interrupts?! Now that sounds interesting (I'm 
>    not saying impossible).

ah shoot, you're right. doing too much and got the two backwards.
ignore my email, I'm going off to sit in the corner with a dunce cap
on..

> 
> WBR, Sergei

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------