Re: HELP: Power Mac G3 PCI Strangeness in 2.2 and 2.3

Re: HELP: Power Mac G3 PCI Strangeness in 2.2 and 2.3

[Grant Erickson]

On 3/3/2000 7:39 PM, Ani Joshi wrote:
>On Fri, 3 Mar 2000, Grant Erickson wrote:
>
>>
>>         for (i = 0; i < 10000000; i++) {
>>                 pci_read_config_word(dp, PCI_VENDOR_ID, &data);
>>                 if (data != 0xFFFF)
>>                         break;
>>                 pci_read_config_word(dp, PCI_DEVICE_ID, &data);
>>                 if (data != 0x0000)
>>                         break;
>>         }
>>
>>         for (i = 0; i < 100000000; i++)
>>                 readl(chip);
>
>	why do you do these loops 100000000 times?

So I can  capture traces on the logic analyzer. Sure, I could just
trigger on a one-time event, but this allows me to do an on-the-fly
capture and get some data.

>> 	if (readl(chip) != 0x00000103)
>> 		printk("Bad Chip ID!\n");
>
>	have you tried printking readl(chip)?  what you're expeciting it
>to be may not be correct or you may have to use a slightly different load
>instruction.  btw, you can use pci_resource_{start,end,len}(pdev, res)
>instead of computing length and all.

Yes. I actually ended up figuring all this out not long after I sent out
the e-mail on Friday.

As reported earlier, after instrumenting the PowerPC machine's PCI bus
with a logic analyzer and running a series of tests, I noticed the
following things:

        1) PCI configuration accesses function normally and within spec
           as evidenced by both correct software operation and waveform
           analysis on the logic analyzer.

        2) With the chip's memory space enable bit activated, read
           accesses to the Chip ID register failed to generate any
           PCI bus traffic of any sort and most definitely resulted
           in failed software operation.

After modifying the Linux kernel to accept PCI devices with the invalid
vendor/device ID combination of 0xffff/0x0000, I noticed something odd
about the way the system PROM (and to a lesser extent Linux) configured
the chip's base addresses. Looking at a number of other devices in the
system:

        grant@marathon% cat /proc/iomem
        81800000-8180001f : PCI device 1022:2000 AMD Lance Ethernet
        81900000-819fffff : PCI device 1022:2000 AMD Lance Ethernet
        82000000-82ffffff : PCI device 1002:4754 ATI Mach 128
        f3000000-f307ffff : PCI device 106b:0010 Apple Heathrow I/O
        fffc0000-ffffffff : PCI device ffff:0000 ???

Clue 1: The address range programmed for the chip looks suspiciously like
the state it might be left in after determining how much address space it
needs.

Before Linux does this address space sizing, it saves the base address.
In the case of my chip, it would have read "0x0" since nothing was
programmed there previously. If Linux does find 0x0, it just leaves the
result of the address sizing there and moves on.

So, that led me to wonder where the addresses in the other devices were
coming from...

I found that even before Linux executes a single line of code, all the
other devices above have valid addresses programmed into their base
address registers. Who's doing this?

I believe that the Open Firmware PROM in the PowerPC system probes all
PCI devices in the system at power up. It then assigns valid base
addresses to those devices with VALID vendor and device IDs.

Because my chip uses an invalid vendor ID of 0xFFFF, Open Firmware must
just ignore it and never programs in a valid base address! Therefore,
when we go and try to access the chip at 0xFFFC0000 (which is outside the
valid PCI address space on PReP/CHRP/PowerMac systems?) we're off probing
never-never land. The PCI controller never assumes the transaction!

So, if I fixup the chip's base address register to something like
0x83000000 beforehand, everything works fine! The problem is that I'm not
sure I can unilaterally guarantee 0x83000000 will be available every time
as I don't know the boot ROM's mapping strategy.

 Grant Erickson                       University of Minnesota Alumni
  o mail:grant@borg.umn.edu                               1996 BSEE
  o http://www.tc.umn.edu/~erick205                       1998 MSEE


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: HELP: Power Mac G3 PCI Strangeness in 2.2 and 2.3

[Geert Uytterhoeven]

On Sun, 5 Mar 2000, Grant Erickson wrote:
> After modifying the Linux kernel to accept PCI devices with the invalid
> vendor/device ID combination of 0xffff/0x0000, I noticed something odd
> about the way the system PROM (and to a lesser extent Linux) configured
> the chip's base addresses. Looking at a number of other devices in the
> system:
>
>         grant@marathon% cat /proc/iomem
>         81800000-8180001f : PCI device 1022:2000 AMD Lance Ethernet
>         81900000-819fffff : PCI device 1022:2000 AMD Lance Ethernet
>         82000000-82ffffff : PCI device 1002:4754 ATI Mach 128
>         f3000000-f307ffff : PCI device 106b:0010 Apple Heathrow I/O
>         fffc0000-ffffffff : PCI device ffff:0000 ???
>
> Clue 1: The address range programmed for the chip looks suspiciously like
> the state it might be left in after determining how much address space it
> needs.

Yes.

> Before Linux does this address space sizing, it saves the base address.
> In the case of my chip, it would have read "0x0" since nothing was
> programmed there previously. If Linux does find 0x0, it just leaves the
> result of the address sizing there and moves on.
>
> So, that led me to wonder where the addresses in the other devices were
> coming from...
>
> I found that even before Linux executes a single line of code, all the
> other devices above have valid addresses programmed into their base
> address registers. Who's doing this?
>
> I believe that the Open Firmware PROM in the PowerPC system probes all
> PCI devices in the system at power up. It then assigns valid base
> addresses to those devices with VALID vendor and device IDs.

OF (should) assign addresses. On my box (CHRP), it does so for _recognized_
devices only.

> Because my chip uses an invalid vendor ID of 0xFFFF, Open Firmware must
> just ignore it and never programs in a valid base address! Therefore,
> when we go and try to access the chip at 0xFFFC0000 (which is outside the
> valid PCI address space on PReP/CHRP/PowerMac systems?) we're off probing
> never-never land. The PCI controller never assumes the transaction!

How difficult is it to make the PCI chip use a valid vendor/device ID? I don't
think you can assume any boot ROM will try to assign a region to it. There's no
way to distinguish a device with an invalid vendor/device ID from a device
that's not even present.

Go read the PCI spec and use valid IDs.

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven ------------- Sony Software Development Center Europe (SDCE)
Geert.Uytterhoeven@sonycom.com ------------------- Sint-Stevens-Woluwestraat 55
Voice +32-2-7248638 Fax +32-2-7262686 ---------------- B-1130 Brussels, Belgium


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Grabbing an Interrupt for a Brain-dead PCI Device

[Grant Erickson]

So, unfortunately, I didn't do the design for this "dumb" PCI device and
neither can I fix its shortcomings in these regards.

Below, we've acknowledged and discovered how to sort of work-around the
problem of the invalid vendor ID and the base address. Now, I've yet
another problem.

The device has an interrupt at INTA#. Unfortunately, the device being dumb
as it is, doesn't utilize the interrupt pin register to advertise that it
does in fact have an interrupt there. So, I assume that once again, the
PROM doesn't give it one.

Is there a way, in Linux, for me to programmatically and manually assign
an interrupt to this device as might have been done in the PROM? I'm
guessing I have to twiddle some registers in the North Bridge, correct?

Thanks,

Grant


On Mon, 6 Mar 2000, Geert Uytterhoeven wrote:
> On Sun, 5 Mar 2000, Grant Erickson wrote:
> > After modifying the Linux kernel to accept PCI devices with the invalid
> > vendor/device ID combination of 0xffff/0x0000, I noticed something odd
> > about the way the system PROM (and to a lesser extent Linux) configured
> > the chip's base addresses. Looking at a number of other devices in the
> > system:
> >
> >         grant@marathon% cat /proc/iomem
> >         81800000-8180001f : PCI device 1022:2000 AMD Lance Ethernet
> >         81900000-819fffff : PCI device 1022:2000 AMD Lance Ethernet
> >         82000000-82ffffff : PCI device 1002:4754 ATI Mach 128
> >         f3000000-f307ffff : PCI device 106b:0010 Apple Heathrow I/O
> >         fffc0000-ffffffff : PCI device ffff:0000 ???
> >
> > Clue 1: The address range programmed for the chip looks suspiciously like
> > the state it might be left in after determining how much address space it
> > needs.
>
> Yes.
>
> > Before Linux does this address space sizing, it saves the base address.
> > In the case of my chip, it would have read "0x0" since nothing was
> > programmed there previously. If Linux does find 0x0, it just leaves the
> > result of the address sizing there and moves on.
> >
> > So, that led me to wonder where the addresses in the other devices were
> > coming from...
> >
> > I found that even before Linux executes a single line of code, all the
> > other devices above have valid addresses programmed into their base
> > address registers. Who's doing this?
> >
> > I believe that the Open Firmware PROM in the PowerPC system probes all
> > PCI devices in the system at power up. It then assigns valid base
> > addresses to those devices with VALID vendor and device IDs.
>
> OF (should) assign addresses. On my box (CHRP), it does so for _recognized_
> devices only.
>
> > Because my chip uses an invalid vendor ID of 0xFFFF, Open Firmware must
> > just ignore it and never programs in a valid base address! Therefore,
> > when we go and try to access the chip at 0xFFFC0000 (which is outside the
> > valid PCI address space on PReP/CHRP/PowerMac systems?) we're off probing
> > never-never land. The PCI controller never assumes the transaction!
>
> How difficult is it to make the PCI chip use a valid vendor/device ID? I don't
> think you can assume any boot ROM will try to assign a region to it. There's no
> way to distinguish a device with an invalid vendor/device ID from a device
> that's not even present.
>
> Go read the PCI spec and use valid IDs.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Grabbing an Interrupt for a Brain-dead PCI Device

[Geert Uytterhoeven]

On Tue, 7 Mar 2000, Grant Erickson wrote:
> So, unfortunately, I didn't do the design for this "dumb" PCI device and
> neither can I fix its shortcomings in these regards.

Hmm... That's bad luck. Then you have to do some things manually (read: in your
own code).

> Below, we've acknowledged and discovered how to sort of work-around the
> problem of the invalid vendor ID and the base address. Now, I've yet
> another problem.
>
> The device has an interrupt at INTA#. Unfortunately, the device being dumb
> as it is, doesn't utilize the interrupt pin register to advertise that it
> does in fact have an interrupt there. So, I assume that once again, the
> PROM doesn't give it one.
>
> Is there a way, in Linux, for me to programmatically and manually assign
> an interrupt to this device as might have been done in the PROM? I'm
> guessing I have to twiddle some registers in the North Bridge, correct?

In most machines, PCI interrupt routing is hardwired on the motherboard.  Hence
you probably don't have to program the North Bridge for that.

Furthermore the interrupt pin register is read-only on normal cards, and
initialized to INTA#. It's not used by the PCI device itself. The interrupt
number register is read-write and is only there so the PROM code (which is
supposed to know the interrupt routing on the motherboard) can fill in the
correct value and applications can read it back later.

I think it's sufficient to let your PCI fixup code fix up the interrupt fields
in the struct pci_dev that corresponds to your weird device.

Good luck!

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Grabbing an Interrupt for a Brain-dead PCI Device

[Michel Lanners]

Hi there,

On   7 Mar, this message from Grant Erickson echoed through cyberspace:
> The device has an interrupt at INTA#. Unfortunately, the device being dumb
> as it is, doesn't utilize the interrupt pin register to advertise that it
> does in fact have an interrupt there. So, I assume that once again, the
> PROM doesn't give it one.

Of course not... how should it know? But....

> Is there a way, in Linux, for me to programmatically and manually assign
> an interrupt to this device as might have been done in the PROM? I'm
> guessing I have to twiddle some registers in the North Bridge, correct?

As far as I know, yes, it depends on the host bridge. However, I don't
think there's any bridge configuration involved here, as the interrupt
pin routing would rather be done in hardware.

If this is in a PowerMac (at least those previous to UMA), then there
is a fixed IRQ per physical slot. All 4 PCI interrupt lines are OR'ed
together per slot.

However, there are other IRQ routing schemes as well... the above being
the easiest one, because the IRQ value can be hardcoded based on the
slot, which you can determine by software.

Plus, there are ways to determine an unknown IRQ by letting the device
generate an interrupt, and seing what IRQ it comes in on. You might
find more info in 'Linux Device Drivers' by Alessandro Rubini.

Michel

-------------------------------------------------------------------------
Michel Lanners                 |  " Read Philosophy.  Study Art.
23, Rue Paul Henkes            |    Ask Questions.  Make Mistakes.
L-1710 Luxembourg              |
email   mlan@cpu.lu            |
http://www.cpu.lu/~mlan        |                     Learn Always. "


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Grabbing an Interrupt for a Brain-dead PCI Device

[Geert Uytterhoeven]

On Tue, 7 Mar 2000, Michel Lanners wrote:
> If this is in a PowerMac (at least those previous to UMA), then there
> is a fixed IRQ per physical slot. All 4 PCI interrupt lines are OR'ed
> together per slot.
>
> However, there are other IRQ routing schemes as well... the above being
> the easiest one, because the IRQ value can be hardcoded based on the
> slot, which you can determine by software.

The most popular routing scheme looks like this:

    #INTA(slot_nr) -> pci_int(slot_nr % 4)
    #INTB(slot_nr) -> pci_int((slot_nr+1) % 4)
    #INTC(slot_nr) -> pci_int((slot_nr+2) % 4)
    #INTD(slot_nr) -> pci_int((slot_nr+3) % 4)

where

    pci_int(0) = #INTW
    pci_int(1) = #INTX
    pci_int(2) = #INTY
    pci_int(3) = #INTZ

Hence if all boards use one interrupt (the first interrupt on a board _must_ be
#INTA), they're evenly spread across the available interrupt sources.

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven ------------- Sony Software Development Center Europe (SDCE)
Geert.Uytterhoeven@sonycom.com ------------------- Sint-Stevens-Woluwestraat 55
Voice +32-2-7248638 Fax +32-2-7262686 ---------------- B-1130 Brussels, Belgium


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Grabbing an Interrupt for a Brain-dead PCI Device

[Matt Porter]

On Wed, Mar 08, 2000 at 09:31:42AM +0100, Geert Uytterhoeven wrote:
>
> On Tue, 7 Mar 2000, Michel Lanners wrote:
> > If this is in a PowerMac (at least those previous to UMA), then there
> > is a fixed IRQ per physical slot. All 4 PCI interrupt lines are OR'ed
> > together per slot.
> >
> > However, there are other IRQ routing schemes as well... the above being
> > the easiest one, because the IRQ value can be hardcoded based on the
> > slot, which you can determine by software.
>
> The most popular routing scheme looks like this:
>
>     #INTA(slot_nr) -> pci_int(slot_nr % 4)
>     #INTB(slot_nr) -> pci_int((slot_nr+1) % 4)
>     #INTC(slot_nr) -> pci_int((slot_nr+2) % 4)
>     #INTD(slot_nr) -> pci_int((slot_nr+3) % 4)
>
> where
>
>     pci_int(0) = #INTW
>     pci_int(1) = #INTX
>     pci_int(2) = #INTY
>     pci_int(3) = #INTZ
>
> Hence if all boards use one interrupt (the first interrupt on a board _must_ be
> #INTA), they're evenly spread across the available interrupt sources.

This is documented in the PCI spec as the suggested method of routing irqs.
We usually call it the standard PCI swizzle, since the interrupts
tumble as they pass through each bridge.  I added this algorithm into
prep_pci.c for MCG boards recently since it is pretty crucial for CPCI
systems that are loaded up with bridges.  There are also plenty of regular
PCI boards that have a P2P and multiple single function devices on board
that also require this routing (old dual Ethernet cards come to mind,
slapping two 21x4x chips and a transparent bridge onboard).

--
Matt Porter
mmporter@home.com

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Grabbing an Interrupt for a Brain-dead PCI Device

[Grant Erickson]

Thanks for all the assistance on this issue thus far. This has been an
interesting trial under the guise of "life is just unnecessarily hard when
hardware designers fail to do things the right way."

My system:
	Power Macintosh G3 300

So, after playing with a few other PCI cards, I find:

                 ---------------------------------------------------------
     Board Slot  |      0      |      1      |      2      |      3      |
                 ---------------------------------------------------------
Pin  Product     |  IRQ   Dev  |  IRQ   Dev  |  IRQ   Dev  |  IRQ    Dev |
                 ---------------------------------------------------------
 A   AMD Lance   |   25    15  |   24    14  |   23    13  |   --     -- |
 A   ATI Rage*   |   --    --  |   --    --  |   --    --  |   22     18 |
 A   Prisa NetFX |   25    15  |   24    14  |   23    13  |   --     -- |
 A** My Device   |   --    15  |   --    14  |   --    13  |   --     -- |
                 ---------------------------------------------------------

  * The ATI Rage is soldered onto the motherboard.
 ** My brain-dead PCI device uses INTA# but doesn't advertise it via the
    PCI_INTERRUPT_PIN register.

So after experimenting with this for a while, I think* that I can assign
the interrupt based on the PCI device the card is assigned.

Also, regarding memory address space resources, is there a routine I can
call that says "Assign this card some resource space because it hasn't
been assigned one yet"?

Thanks again,

Grant


On Tue, 7 Mar 2000, Michel Lanners wrote:
> On   7 Mar, this message from Grant Erickson echoed through cyberspace:
> > The device has an interrupt at INTA#. Unfortunately, the device being dumb
> > as it is, doesn't utilize the interrupt pin register to advertise that it
> > does in fact have an interrupt there. So, I assume that once again, the
> > PROM doesn't give it one.
>
> Of course not... how should it know? But....
>
> > Is there a way, in Linux, for me to programmatically and manually assign
> > an interrupt to this device as might have been done in the PROM? I'm
> > guessing I have to twiddle some registers in the North Bridge, correct?
>
> As far as I know, yes, it depends on the host bridge. However, I don't
> think there's any bridge configuration involved here, as the interrupt
> pin routing would rather be done in hardware.
>
> If this is in a PowerMac (at least those previous to UMA), then there
> is a fixed IRQ per physical slot. All 4 PCI interrupt lines are OR'ed
> together per slot.
>
> However, there are other IRQ routing schemes as well... the above being
> the easiest one, because the IRQ value can be hardcoded based on the
> slot, which you can determine by software.
>
> Plus, there are ways to determine an unknown IRQ by letting the device
> generate an interrupt, and seing what IRQ it comes in on. You might
> find more info in 'Linux Device Drivers' by Alessandro Rubini.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Grabbing an Interrupt for a Brain-dead PCI Device

[Geert Uytterhoeven]

On Wed, 8 Mar 2000, Grant Erickson wrote:
> Also, regarding memory address space resources, is there a routine I can
> call that says "Assign this card some resource space because it hasn't
> been assigned one yet"?

Yes there is, somewhere in drivers/pci/. I don't know yet whether it works.

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Grabbing an Interrupt for a Brain-dead PCI Device

[Michel Lanners]

On   8 Mar, this message from Geert Uytterhoeven echoed through cyberspace:
>> Also, regarding memory address space resources, is there a routine I can
>> call that says "Assign this card some resource space because it hasn't
>> been assigned one yet"?
>
> Yes there is, somewhere in drivers/pci/. I don't know yet whether it works.

I doubt it will work on a PowerMac; see what the resource 'checking'
code finds as parent resources (r is a mem resource, pr is its parent
resource):

Mar  9 07:24:29 piglet kernel:    Resources: r.start=f2000400, r.end=f200043f,
pr.start=00000000, pr.end=ffffffff.

As far as I can tell, parent resource would be the memory region held
by the parent bridge, out of which this region was allocated. A bit
huge, this region, isn't it?

Then again, Apple's host bridges might not conform entirely to the PCI
spec in the way they advertise their resources available for
assignment...

Those with the PCI spec at hand out there, is there such a thing as
resource registers in bridges (be it P2P or host) that inform about
what is available to devices behind this particular bridge?

Maybe there's another job for the fixup code...

Michel

-------------------------------------------------------------------------
Michel Lanners                 |  " Read Philosophy.  Study Art.
23, Rue Paul Henkes            |    Ask Questions.  Make Mistakes.
L-1710 Luxembourg              |
email   mlan@cpu.lu            |
http://www.cpu.lu/~mlan        |                     Learn Always. "


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Grabbing an Interrupt for a Brain-dead PCI Device

[Gabriel Paubert]

> I doubt it will work on a PowerMac; see what the resource 'checking'
> code finds as parent resources (r is a mem resource, pr is its parent
> resource):
>
> Mar  9 07:24:29 piglet kernel:    Resources: r.start=f2000400, r.end=f200043f,
> pr.start=00000000, pr.end=ffffffff.
>
> As far as I can tell, parent resource would be the memory region held
> by the parent bridge, out of which this region was allocated. A bit
> huge, this region, isn't it?

Indeed...

> Then again, Apple's host bridges might not conform entirely to the PCI
> spec in the way they advertise their resources available for
> assignment...

They probably do, at least OF should get it right in the range properties
of every bus (that's what I have on a very buggy OF for PreP Motorola
boards):

ranges                   01000000 00000000 00000000 80000000 00010000
                         01000000 00000000 01000000 81000000 3e800000
                         02000000 00000000 00000000 c0000000 3c000000
			 c172616e 67657386 0005c32c

the first 2 lines indicate how the I/O space is mapped from the processor
and the 3rd the PCI memory  space. The last line is junk...

> Those with the PCI spec at hand out there, is there such a thing as
> resource registers in bridges (be it P2P or host) that inform about
> what is available to devices behind this particular bridge?

P2P do have them, host bridges do not have them although they often have
chip specific registers which allow t ochoose between several options or
to program where the I/O space.

> Maybe there's another job for the fixup code...

Indeed, actually /proc/iomem on Intel includes even the RAM ! What is
needed is to include one hierarchy level to define the physical addresses
which the bridge forward to PCI (I started a patch but never finished it).

	Gabriel.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/