Help enabling PCI interrupts on Dell/SMP and Sun/SMP systems.

Help enabling PCI interrupts on Dell/SMP and Sun/SMP systems.

[Alan Kilian]

    Folks,

	This group was instrumental in helping me get my first-ever
	linux/PCI-bus device driver working last year, and I'm back for
	some more help if you are willing.

	I have a PCI card that generates an interrupt when it completes
	a DMA transfer to the PCs RAM.

	This works just fine on a Dell 4400 running 2.6.10-1.766_FC3

	When I try to run the driver on a Dell 2300 FC2/2.6.5-1.358smp
	or a Sun W2100Z running FC2/2.6.10-1.14_FC2smp I can see the
	DMA-done bit set in the device, but my interrupt service routine
	never gets called.

	On the Sun, I booted with "noapic" option, and it booted OK,
	but then when my device generated an interrupt, there was a
	kernel message about Disabling IRQ #5 and the system was hung
	solidly.

	I think this has something to do with the different interrupt
	hardware on the more advanced servers compared to my desktop
	Dell 4400, and I somehow need to "enable" the IOAPIC system
	so that my interrupt gets through to my service routine, but I
	don't know how.

	I tried grepping through the kernel/drivers source, and I didn't
	find anything that jumped out at me.

	The Rubini drivers book didn't help in this area either, 
	although it's a wonderful book in other areas.

	I can post source somewhere if it will help.

	I can also post the essential bits from /var/log/messages about
	all the incredibly complicated IOAPIC configuration stuff.

	Thank you for your past help, and thank you in advance for any
	tips you can provide.

				-Alan

-- 
- Alan Kilian <kilian(at)bobodyne.com>

Re: Help enabling PCI interrupts on Dell/SMP and Sun/SMP systems.

[linux-os]

On Wed, 23 Feb 2005, Alan Kilian wrote:

 	call	pci_enable_device(dev)
 	... before you use the IRQ in dev->irq.

 	The reported IRQ is bogus until you make that
 	call. It's a reported BUG, probably won't
 	ever get fixed because it's considered a
 	feature.

 	Also, make sure that your .config for the Dell looks
 	something like:

 	CONFIG_X86_IO_APIC=y
 	CONFIG_X86_LOCAL_APIC=y
 	CONFIG_PCI=y
 	# CONFIG_PCI_GOBIOS is not set
 	# CONFIG_PCI_GODIRECT is not set
 	CONFIG_PCI_GOANY=y
 	CONFIG_PCI_BIOS=y
 	CONFIG_PCI_DIRECT=y

>
>
>    Folks,
>
> 	This group was instrumental in helping me get my first-ever
> 	linux/PCI-bus device driver working last year, and I'm back for
> 	some more help if you are willing.
>
> 	I have a PCI card that generates an interrupt when it completes
> 	a DMA transfer to the PCs RAM.
>
> 	This works just fine on a Dell 4400 running 2.6.10-1.766_FC3
>
> 	When I try to run the driver on a Dell 2300 FC2/2.6.5-1.358smp
> 	or a Sun W2100Z running FC2/2.6.10-1.14_FC2smp I can see the
> 	DMA-done bit set in the device, but my interrupt service routine
> 	never gets called.
>
> 	On the Sun, I booted with "noapic" option, and it booted OK,
> 	but then when my device generated an interrupt, there was a
> 	kernel message about Disabling IRQ #5 and the system was hung
> 	solidly.
>
> 	I think this has something to do with the different interrupt
> 	hardware on the more advanced servers compared to my desktop
> 	Dell 4400, and I somehow need to "enable" the IOAPIC system
> 	so that my interrupt gets through to my service routine, but I
> 	don't know how.
>
> 	I tried grepping through the kernel/drivers source, and I didn't
> 	find anything that jumped out at me.
>
> 	The Rubini drivers book didn't help in this area either,
> 	although it's a wonderful book in other areas.
>
> 	I can post source somewhere if it will help.
>
> 	I can also post the essential bits from /var/log/messages about
> 	all the incredibly complicated IOAPIC configuration stuff.
>
> 	Thank you for your past help, and thank you in advance for any
> 	tips you can provide.
>
> 				-Alan
>
> -- 
> - Alan Kilian <kilian(at)bobodyne.com>
>

Cheers,
Dick Johnson
Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips).
  Notice : All mail here is now cached for review by Dictator Bush.
                  98.36% of all statistics are fiction.

Re: Help enabling PCI interrupts on Dell/SMP and Sun/SMP systems.

[Alan Kilian]

> On Wed, 23 Feb 2005, Dick Johnson wrote:
> 
>  	call	pci_enable_device(dev)
>  	... before you use the IRQ in dev->irq.
> 
>  	The reported IRQ is bogus until you make that
>  	call. It's a reported BUG, probably won't
>  	ever get fixed because it's considered a
>  	feature.
> 
>  	Also, make sure that your .config for the Dell looks
>  	something like:
> 
>  	CONFIG_X86_IO_APIC=y
>  	CONFIG_X86_LOCAL_APIC=y
>  	CONFIG_PCI=y
>  	# CONFIG_PCI_GOBIOS is not set
>  	# CONFIG_PCI_GODIRECT is not set
>  	CONFIG_PCI_GOANY=y
>  	CONFIG_PCI_BIOS=y
>  	CONFIG_PCI_DIRECT=y


    Dick,

	Thanks for the quick reply.

	1) I call pci_enable_device(dev) immediatly after I call
	   dev = pci_find_device(0x1492, PCI_ANY_ID, dev);

	2) I have verified all the CONFIG settings you suggested.

	Here is `cat /proc/interrupts` on my working dell:

          	 CPU0
  	  0:   16891629          XT-PIC  timer
  	  1:         10          XT-PIC  i8042
  	  2:          0          XT-PIC  cascade
  	  3:          2          XT-PIC  parport0
  	  5:        764          XT-PIC  sse
  	  7:        422          XT-PIC  ohci_hcd
  	  8:          1          XT-PIC  rtc
  	  9:          0          XT-PIC  acpi
 	 11:      35198          XT-PIC  eth0
 	 12:         66          XT-PIC  i8042
 	 14:      42769          XT-PIC  ide0
 	 15:     151569          XT-PIC  ide1
	NMI:          0
	ERR:          0

	My driver is called "sse" and is interrupting at IRQ #5

	Here is `cat /proc/interrupts` on my non-working Sun:

           	CPU0       CPU1
  	  0:    7302649    7417311    IO-APIC-edge  timer
  	  5:          0          0    IO-APIC-edge  sse
  	  8:          0          1    IO-APIC-edge  rtc
  	  9:          0          0   IO-APIC-level  acpi
 	 15:          1        478    IO-APIC-edge  ide1
	169:          0         30   IO-APIC-level  aic79xx
	177:      13991      18084   IO-APIC-level  aic79xx
	185:          0          3   IO-APIC-level  ehci_hcd
	193:          0         26   IO-APIC-level  ohci_hcd
	201:          0         21   IO-APIC-level  ohci_hcd
	209:     167119         87   IO-APIC-level  eth0
	NMI:          0          0
	LOC:   14719159   14719203
	ERR:          0
	MIS:          0

	It appears that the card is also interrupting at IRQ#5

	There is an interesting message on the Sun in /var/log/messages:

Feb 23 14:01:26 sunw1200z kernel: sse: no version magic, tainting
kernel.
Feb 23 14:01:26 sunw1200z kernel: SSE: Found a DeCypher card.
Feb 23 14:01:26 sunw1200z kernel: ACPI: PCI interrupt 0000:13:03.0[A] ->
GSI 36 (level, low) -> IRQ 217
Feb 23 14:01:26 sunw1200z kernel: interrupting on line 5
Feb 23 14:01:26 sunw1200z kernel: SSE: bar[0] From 0xd2806000 to
0xd2806fff F=0x200 MEMORY space
Feb 23 14:01:26 sunw1200z kernel: SSE: bar[1] From 0xd2800000 to
0xd2801fff F=0x200 MEMORY space
Feb 23 14:01:26 sunw1200z kernel: SSE: bar[2] From 0xd2000000 to
0xd27fffff F=0x200 MEMORY space
Feb 23 14:01:26 sunw1200z kernel: SSE: bar[0] mybase = 0xf889a000 size =
0x00001000 D'4096
Feb 23 14:01:26 sunw1200z kernel: SSE: bar[1] mybase = 0xf889c000 size =
0x00002000 D'8192
Feb 23 14:01:26 sunw1200z kernel: SSE: bar[2] mybase = 0xf8b00000 size =
0x00800000 D'8388608
Feb 23 14:01:26 sunw1200z kernel: pci_alloc_consistent returned
0xf0ded000
Feb 23 14:01:26 sunw1200z kernel: sse_read_bus_buffer 0x30ded000
Feb 23 14:01:26 sunw1200z kernel: request_irq() returned 0
Feb 23 14:01:26 sunw1200z kernel: SSE device_id 3, Rev 4
Feb 23 14:01:26 sunw1200z kernel: SSE Before: intstatus = 0x00000000
Feb 23 14:01:26 sunw1200z kernel: SSE Before: intstatus = 0x00000000
Feb 23 14:01:26 sunw1200z kernel: SSE: End of card attachment. Number of
cards = 1
Feb 23 14:01:26 sunw1200z kernel: Iterating through the softp
structures...
Feb 23 14:01:26 sunw1200z kernel: Card at softp->mem_reg1 0xf889a000
minor = 0


	The interesting bits seem to be these two lines:

	kernel: SSE: Found a DeCypher card.
	kernel: ACPI: PCI interrupt 0000:13:03.0[A] -> GSI 36 (level, low) ->
IRQ 217

	The first message is in my driver after pci_find_device()
	The second is from when I do pci_enable_device(dev);

	Can you decode the mysterious ACPI message?

			-Alan

-- 
- Alan Kilian <kilian(at)bobodyne.com>

Re: Help enabling PCI interrupts on Dell/SMP and Sun/SMP systems.

[Peter Chubb]

>>>>> "Alan" == Alan Kilian <kilian@bobodyne.com> writes:






Alan> 	kernel: SSE: Found a DeCypher card.  kernel: ACPI: PCI
Alan> interrupt 0000:13:03.0[A] -> GSI 36 (level, low) -> IRQ 217

If ACPI has set this device up to use interrupt 217, why are you
registering it on IRQ 5?

--
Dr Peter Chubb  http://www.gelato.unsw.edu.au  peterc AT gelato.unsw.edu.au
The technical we do immediately,  the political takes *forever*

Re: Help enabling PCI interrupts on Dell/SMP and Sun/SMP systems.

[Alan Kilian]

> Alan> 	kernel: SSE: Found a DeCypher card.  kernel: ACPI: PCI
> Alan> interrupt 0000:13:03.0[A] -> GSI 36 (level, low) -> IRQ 217
> 
> If ACPI has set this device up to use interrupt 217, why are you
> registering it on IRQ 5?
> 

    Peter,

	Maybe that's it.

	I ask the card which interrupt line it was given at boot-time:

	pci_read_config_byte(dev, PCI_INTERRUPT_LINE,
	                     &softp->interrupt_line);

	Then I request an IRQ:

	request_irq(softp->interrupt_line, sseintr, 
                    SA_INTERRUPT, "sse", softp);

	I don't know who exactly is assigning IRQ 217, and I don't know
	where I could read that value.

	Are you suggesting I should call request_irq(217, sseintr, 
                    SA_INTERRUPT, "sse", softp);

	Maybe you are on to the problem.

	I found out that if I boot with   "pci-noapic" that everyting
	works on the Sun machine!!!

		-Alan

-- 
- Alan Kilian <kilian(at)bobodyne.com>

Re: Help enabling PCI interrupts on Dell/SMP and Sun/SMP systems.

[Robert Hancock]

Alan Kilian wrote:
> 	kernel: SSE: Found a DeCypher card.
> 	kernel: ACPI: PCI interrupt 0000:13:03.0[A] -> GSI 36 (level, low) ->
> IRQ 217
> 
> 	The first message is in my driver after pci_find_device()
> 	The second is from when I do pci_enable_device(dev);
> 
> 	Can you decode the mysterious ACPI message?

Looks like you're requesting the wrong interrupt, 217 is the one your 
device is actually on. You always have to request the interrupt listed 
in the PCI device structure. If you're looking at your PCI device's 
configuration registers to get the IRQ to request, that's wrong, since 
that is configured by the BIOS assuming PIC IRQ routing, but APIC IRQ 
routing is entirely different.

Re: Help enabling PCI interrupts on Dell/SMP and Sun/SMP systems.

[Roland Dreier]

    Alan> I ask the card which interrupt line it was given at
    Alan> boot-time:

    Alan> 	pci_read_config_byte(dev, PCI_INTERRUPT_LINE, &softp->interrupt_line);

    Alan> Then I request an IRQ:

    Alan> 	request_irq(softp->interrupt_line, sseintr, SA_INTERRUPT, "sse", softp);

Don't do that.  The kernel may need you to use a different interrupt
number than you read from the PCI config header.  Use dev->irq, as in

	request_irq(dev->irq, sseintr, SA_SHIRQ | SA_INTERRUPT, "sse", softp);

Also, make sure that you do pci_enable_device(dev) before you look at
dev->irq, since the field will not be initialized until you do that.

 - R.

Re: Help enabling PCI interrupts on Dell/SMP and Sun/SMP systems.

[Robert Hancock]

Alan Kilian wrote:
> 	Maybe that's it.
> 
> 	I ask the card which interrupt line it was given at boot-time:
> 
> 	pci_read_config_byte(dev, PCI_INTERRUPT_LINE,
> 	                     &softp->interrupt_line);
> 
> 	Then I request an IRQ:
> 
> 	request_irq(softp->interrupt_line, sseintr, 
>                     SA_INTERRUPT, "sse", softp);

Yeah, that's wrong. Should be request_irq(dev->irq, ... )

PCI_INTERRUPT_LINE is assigned by the BIOS and has nothing to do with 
the routing used in APIC mode. That's why it works with noapic mode 
since then the routing is the same as the BIOS assumed.

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

Re: Help enabling PCI interrupts on Dell/SMP and Sun/SMP systems.

[Alan Kilian]

On Wed, 2005-02-23 at 15:46 -0800, Roland Dreier wrote:
>     Alan> 	pci_read_config_byte(dev, PCI_INTERRUPT_LINE, &softp->interrupt_line);
>     Alan> 	request_irq(softp->interrupt_line, sseintr, SA_INTERRUPT, "sse", softp);
> 
> Don't do that.  The kernel may need you to use a different interrupt
> number than you read from the PCI config header.  Use dev->irq, as in
> 
> 	request_irq(dev->irq, sseintr, SA_SHIRQ | SA_INTERRUPT, "sse", softp);

    Roland, 

	You are the best. Thank you so much for this information.

    	That was the ticket.

    	request_irq(dev->irq, sseintr, SA_SHIRQ | SA_INTERRUPT, 
	            "sse", softp);

    	It works on all machines now without adding "pci=noapci"

    	I'm running again!!!

                                     -Alan

-- 
- Alan Kilian <kilian(at)bobodyne.com>

Re: Help enabling PCI interrupts on Dell/SMP and Sun/SMP systems.

[linux-os]

Where are you getting IRQ5 from? You can't "hard-code" interrupts on
PCI.

> 	kernel: ACPI: PCI interrupt 0000:13:03.0[A] -> GSI 36 (level, low) ->
> IRQ 217
^^^^^^^^^___________ This is your IRQ

It should be in dev->irq AFTER it's enabled.
[SNIPPED...]


Cheers,
Dick Johnson
Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips).
  Notice : All mail here is now cached for review by Dictator Bush.
                  98.36% of all statistics are fiction.