Failure to detect PCI card

Failure to detect PCI card

[Peter LaDow]

I have a PCI card (a Netgear FA331, vendor:device 100b:0020) that is
failing to be detected by our PPC platform.  This device works just
fine in a PC, and other cards work just fine in the same PCI slot (we
have an Intel 82540EM based card that works).

But for some reason, neither u-boot nor the kernel detect this card.
Any ideas why this might be?

Thanks,
Pete

Re: Failure to detect PCI card

[Anatolij Gustschin]

On Mon, 5 Aug 2013 10:58:01 -0700
Peter LaDow <petela@gocougs.wsu.edu> wrote:

> I have a PCI card (a Netgear FA331, vendor:device 100b:0020) that is
> failing to be detected by our PPC platform.  This device works just
> fine in a PC, and other cards work just fine in the same PCI slot (we
> have an Intel 82540EM based card that works).
> 
> But for some reason, neither u-boot nor the kernel detect this card.
> Any ideas why this might be?

Maybe this card needs bigger delay to respond after PCI reset. You can
try to re-build U-Boot with defined CONFIG_PCI_BOOTDELAY. Use 1000
for CONFIG_PCI_BOOTDELAY in the first step and if detection works,
try to decrease this value.

HTH,

Anatolij

Re: Failure to detect PCI card

[Peter LaDow]

On Mon, Aug 5, 2013 at 11:25 AM, Anatolij Gustschin <agust@denx.de> wrote:
> Maybe this card needs bigger delay to respond after PCI reset. You can
> try to re-build U-Boot with defined CONFIG_PCI_BOOTDELAY. Use 1000
> for CONFIG_PCI_BOOTDELAY in the first step and if detection works,
> try to decrease this value.

That didn't work.  But perhaps the problem is because these are older
PCI parts, and don't implement ECAM (only the older CAM)?   I don't
think u-boot limits itself...

Thanks,
Pete

Re: Failure to detect PCI card

[David Hawkins]

Hi Pete,

> On Mon, Aug 5, 2013 at 11:25 AM, Anatolij Gustschin <agust@denx.de> wrote:
>> Maybe this card needs bigger delay to respond after PCI reset. You can
>> try to re-build U-Boot with defined CONFIG_PCI_BOOTDELAY. Use 1000
>> for CONFIG_PCI_BOOTDELAY in the first step and if detection works,
>> try to decrease this value.
>
> That didn't work.  But perhaps the problem is because these are older
> PCI parts, and don't implement ECAM (only the older CAM)?   I don't
> think u-boot limits itself...

Since you're looking for suggestions (apologies if these are
"too obvious");

1. Have you checked the power supplies on the PCI board?

    PCI boards can be powered from 3.3V or 5V, or both. I've had
    old PCs that only supply one or the other rail, and various
    evaluation boards that only supply 3.3V.

    If you can put together a "working" x86 setup that detects the
    board, then you could poke around and see what voltages exist
    on some of the decoupling components, then plug it into your
    real system, and see what voltages you measure there.

2. Have you probed the PCI bus using a bus analyzer or scope?

    If you have a PCI bus analyzer (or can find someone with one),
    plug it in and see what happens at power-on (there should be
    configuration cycles).

    At a minimum, if you have a 'scope, see if the PCI configuration
    space access handshakes are active during power-on.

3. Is debugging this PCI card worth your time?

    Sometimes the "solution" involves tossing old hardware in
    the trash.

Cheers,
Dave

Re: Failure to detect PCI card

[Peter LaDow]

On Mon, Aug 5, 2013 at 1:27 PM, David Hawkins <dwh@ovro.caltech.edu> wrote:
> 1. Have you checked the power supplies on the PCI board?
>
>    PCI boards can be powered from 3.3V or 5V, or both. I've had
>    old PCs that only supply one or the other rail, and various
>    evaluation boards that only supply 3.3V.
>
>    If you can put together a "working" x86 setup that detects the
>    board, then you could poke around and see what voltages exist
>    on some of the decoupling components, then plug it into your
>    real system, and see what voltages you measure there.

These are universal boards.  Our board does only support only 3V3 (and
is slotted as such).

> 2. Have you probed the PCI bus using a bus analyzer or scope?
>
>    If you have a PCI bus analyzer (or can find someone with one),
>    plug it in and see what happens at power-on (there should be
>    configuration cycles).
>
>    At a minimum, if you have a 'scope, see if the PCI configuration
>    space access handshakes are active during power-on.

Hmm...I do have one.  But I can't get both the analyzer and the card
in the system in at the same time.

> 3. Is debugging this PCI card worth your time?
>
>    Sometimes the "solution" involves tossing old hardware in
>    the trash.

Well, this is part of the ongoing work regarding the incoming PCI
memory corruption.  We are going down the path of abandoning the 82540
for our platform because we can't seem to track down the corruption.
So we are looking at other chipsets which we can purchase, which
include this National (now TI) chipset on the Netgear FA331.  If we
could find a PCI (_not_ PCIe) card to test with that seems to work...

Interestingly, I have an older 3com 3C905TXM exhibiting the same
behavior.  Both of these are older cards (they even came with drivers
on--gasp--floppies!).  Maybe the lack of a 5V supply is an issue...

Thanks,
Pete

Re: Failure to detect PCI card

[David Hawkins]

Hi Pete,

>> 1. Have you checked the power supplies on the PCI board?
>>
>>     PCI boards can be powered from 3.3V or 5V, or both. I've had
>>     old PCs that only supply one or the other rail, and various
>>     evaluation boards that only supply 3.3V.
>>
>>     If you can put together a "working" x86 setup that detects the
>>     board, then you could poke around and see what voltages exist
>>     on some of the decoupling components, then plug it into your
>>     real system, and see what voltages you measure there.
>
> These are universal boards.  Our board does only support only 3V3 (and
> is slotted as such).

Ok.

>> 2. Have you probed the PCI bus using a bus analyzer or scope?
>>
>>     If you have a PCI bus analyzer (or can find someone with one),
>>     plug it in and see what happens at power-on (there should be
>>     configuration cycles).
>>
>>     At a minimum, if you have a 'scope, see if the PCI configuration
>>     space access handshakes are active during power-on.
>
> Hmm...I do have one.  But I can't get both the analyzer and the card
> in the system in at the same time.

My analyzer has an extender card that you first plug in, and then
plug the board into that ... any chance someone in your organization
has one of those cards? Alternatively, confirm the board works in
a machine that has more than one slot, and if it does not, use the
analyzer to see what is happening.

>> 3. Is debugging this PCI card worth your time?
>>
>>     Sometimes the "solution" involves tossing old hardware in
>>     the trash.
>
> Well, this is part of the ongoing work regarding the incoming PCI
> memory corruption.

Ok, I recall seeing that thread.

> We are going down the path of abandoning the 82540
> for our platform because we can't seem to track down the corruption.
> So we are looking at other chipsets which we can purchase, which
> include this National (now TI) chipset on the Netgear FA331.  If we
> could find a PCI (_not_ PCIe) card to test with that seems to work...
>
> Interestingly, I have an older 3com 3C905TXM exhibiting the same
> behavior.  Both of these are older cards (they even came with drivers
> on--gasp--floppies!).  Maybe the lack of a 5V supply is an issue...

If you're looking for a PCI target that you can completely control,
then if you have an "FPGA guy" in your company, perhaps he can
dig up a low-cost PCI card that you can configure as a PCI master
to emulate the functions of a network card.

Actually, before going down that route, I would get a PCI extender
that you can use to trace the traffic with your board. Does the
network card use 33MHz or 66MHz?

Cheers,
Dave

Re: Failure to detect PCI card

[David Hawkins]

Hi Pete,


> Actually, before going down that route, I would get a PCI extender
> that you can use to trace the traffic with your board. Does the
> network card use 33MHz or 66MHz?

I wonder if something like this board:

http://www.logicsupply.com/products/pci122_dflex

can be used to make a single PCI slot into two PCI slots. You
could then plug your analyzer into the second slot.

For $30 it might be worth a shot ...

Cheers,
Dave

Re: Failure to detect PCI card

[Peter LaDow]

On Mon, Aug 5, 2013 at 2:08 PM, David Hawkins <dwh@ovro.caltech.edu> wrote:
> My analyzer has an extender card that you first plug in, and then
> plug the board into that ... any chance someone in your organization
> has one of those cards? Alternatively, confirm the board works in
> a machine that has more than one slot, and if it does not, use the
> analyzer to see what is happening.

Mine has an extender as well, but it is a 5V only extender (such as
the one you pointed out).  I'll ask the HW guys if they have a 3V
somewhere.

Now, I'm not a PCI expoert, but, I can do PCI transactions via u-boot.
 And probing the PCI space via u-boot reveals nothing on 00:14.0 (the
place where our 82540 typically shows up).  Indeed, it returns all
FF's, so I presume that means nothing is there.  And enabling PCI_ENUM
is u-boot causes an exception when I try 'pci enum' (with an older
version of u-boot) or hangs u-boot (in 2013.07).

> If you're looking for a PCI target that you can completely control,
> then if you have an "FPGA guy" in your company, perhaps he can
> dig up a low-cost PCI card that you can configure as a PCI master
> to emulate the functions of a network card.

Well, we do have an FPGA hanging off the bus.  But getting an RTL guy
to do this....hmmm.  I'll have to explore more.

> Actually, before going down that route, I would get a PCI extender
> that you can use to trace the traffic with your board. Does the
> network card use 33MHz or 66MHz?

It's at 33MHz.  I just need to find a 3V extender somewhere...

Thanks,
Pete

Re: Failure to detect PCI card

[Peter LaDow]

On Mon, Aug 5, 2013 at 1:27 PM, David Hawkins <dwh@ovro.caltech.edu> wrote:
> 2. Have you probed the PCI bus using a bus analyzer or scope?

Ok.  I managed to find someone with a bus extender, and have connected
both our analyzer and card.

With a working card (the 82540EM based card), I see configuration
reads on 0x00040000, 0x00080000, and 0x00100000 succeed.  The 4 and 8
correspond to the IDSEL's for our on-board ethernet (also 82540 based)
and our FPGA, respectively (these configuration addresses match the
IDSEL on our schematic, and these accesses are present when no card is
plugged into the expansion slot).  The 0x10 corresponds with the 82540
based network card.  All the vendor/device id's make sense.

However, replacing the 82540 based card with either a 3com 3C905TXM or
the Netgear FA331, there is no response on the 0x10 IDSEL line.  Thus
it appears these cards are NOT responding to configuration reads.  I
think I have to break out the scope and probe around for the 5V/3V.  I
wouldn't think that the supplies are the issue since these are
universal cards, and as I understand it either 3V3 _or_ 5V can be
present, so these should work fine.  Yet they aren't responding to PCI
configuration cycles.

I'll have to do more digging.

Thanks,
Pete

Re: Failure to detect PCI card

[Peter LaDow]

On Mon, Aug 5, 2013 at 4:07 PM, Peter LaDow <petela@gocougs.wsu.edu> wrote:
> However, replacing the 82540 based card with either a 3com 3C905TXM or
> the Netgear FA331, there is no response on the 0x10 IDSEL line.  Thus
> it appears these cards are NOT responding to configuration reads.  I
> think I have to break out the scope and probe around for the 5V/3V.  I
> wouldn't think that the supplies are the issue since these are
> universal cards, and as I understand it either 3V3 _or_ 5V can be
> present, so these should work fine.  Yet they aren't responding to PCI
> configuration cycles.

A few more notes.  I tried a variety of other cards, such as a PCI
modem (WinMode, ack), a WinTV card, and a PCI based 802.11 card.  All
of them enumerate perfectly.

Perhaps it is a BIOS option ROM like you suggested earlier.  The
3c90xC reference manual I found
(http://people.freebsd.org/~wpaul/3Com/3c90xc.pdf) mentions an option
ROM (and there is an Atmel part stuffed).  I can't find any technical
information on the FA331 (yet), so I don't know about it.

But regardless, wouldn't enumeration have to occur before any option
ROM could even be used?

Thanks,
Pete

Re: Failure to detect PCI card

[David Hawkins]

Hi Pete,

> On Mon, Aug 5, 2013 at 4:07 PM, Peter LaDow <petela@gocougs.wsu.edu> wrote:
>> However, replacing the 82540 based card with either a 3com 3C905TXM or
>> the Netgear FA331, there is no response on the 0x10 IDSEL line.  Thus
>> it appears these cards are NOT responding to configuration reads.  I
>> think I have to break out the scope and probe around for the 5V/3V.  I
>> wouldn't think that the supplies are the issue since these are
>> universal cards, and as I understand it either 3V3 _or_ 5V can be
>> present, so these should work fine.  Yet they aren't responding to PCI
>> configuration cycles.

It depends on what you define "Universal PCI" as :)

My definition is that the board works with either 5V or 3.3V PCI
signals. However, that definition does not tell you where the device
draws power from.

I suspect the lack of either the 5V or 3.3V power rail to the card
might be the problem.

Did you probe the PCI edge connect to see what supplies were present?

> A few more notes.  I tried a variety of other cards, such as a PCI
> modem (WinMode, ack), a WinTV card, and a PCI based 802.11 card.  All
> of them enumerate perfectly.

These could all be 3.3V cards ...

> Perhaps it is a BIOS option ROM like you suggested earlier.  The
> 3c90xC reference manual I found
> (http://people.freebsd.org/~wpaul/3Com/3c90xc.pdf) mentions an option
> ROM (and there is an Atmel part stuffed).  I can't find any technical
> information on the FA331 (yet), so I don't know about it.
>
> But regardless, wouldn't enumeration have to occur before any option
> ROM could even be used?

I've never had a card with an option ROM, so can't comment. I'd have
to look at the PCI spec. If you want me to look, let me know.

Cheers,
Dave

Re: Failure to detect PCI card

[Peter LaDow]

On Mon, Aug 5, 2013 at 7:01 PM, David Hawkins <dwh@ovro.caltech.edu> wrote:
> I suspect the lack of either the 5V or 3.3V power rail to the card
> might be the problem.
>
> Did you probe the PCI edge connect to see what supplies were present?

For those that are interested, we did figure out what was going on.
Turns out that the clock buffer driving the PCI connector was, well,
less than adequate.  With some cards, the load on the clock line was
large enough that the clock was in horrible shape.  Fixing the clock
line and the card that failed to be recognized started working.  For
the other cards that worked, the load on the clock line was
significantly less, but the clock was still marginal.

Anyway, turned out to be a hardware issue.  Thanks to all that helped!

Pete

Re: Failure to detect PCI card

[David Hawkins]

Hi Pete,

> For those that are interested, we did figure out what was going on.
> Turns out that the clock buffer driving the PCI connector was, well,
> less than adequate.  With some cards, the load on the clock line was
> large enough that the clock was in horrible shape.  Fixing the clock
> line and the card that failed to be recognized started working.  For
> the other cards that worked, the load on the clock line was
> significantly less, but the clock was still marginal.
>
> Anyway, turned out to be a hardware issue.  Thanks to all that helped!

Thanks for posting that you solved the issue.

The old engineering debug mantra ran true; check the power, then
clocks, then debug :)

Cheers,
Dave

Re: Failure to detect PCI card

[Benjamin Herrenschmidt]

On Thu, 2013-08-08 at 13:31 -0700, Peter LaDow wrote:
> For those that are interested, we did figure out what was going on.
> Turns out that the clock buffer driving the PCI connector was, well,
> less than adequate.  With some cards, the load on the clock line was
> large enough that the clock was in horrible shape.  Fixing the clock
> line and the card that failed to be recognized started working.  For
> the other cards that worked, the load on the clock line was
> significantly less, but the clock was still marginal.
> 
> Anyway, turned out to be a hardware issue.  Thanks to all that helped!

Do that help with e1000 as well ? IE. A bad clock might have caused
malfunctions of the DMA for example...

Cheers,
Ben.

Re: Failure to detect PCI card

[Peter LaDow]

On Sun, Aug 11, 2013 at 7:32 PM, Benjamin Herrenschmidt
<benh@kernel.crashing.org> wrote:
> Do that help with e1000 as well ? IE. A bad clock might have caused
> malfunctions of the DMA for example...

No, it didn't help with the e1000.  In fact, we have separate clocks
for the on-board components.  This particular case was with our
expansion slot having a crappy clock buffer driving the PCI slot.

I'm still digging into the e1000 driver trying to find something.  But
right now we are working on a replacement for the 82540 (which has
been EOL'd anyhow, so it needs to be done, we'd just prefer it be
later rather than sooner).

Thanks,
Pete

Re: Failure to detect PCI card

[Anatolij Gustschin]

On Mon, 5 Aug 2013 16:14:29 -0700
Peter LaDow <petela@gocougs.wsu.edu> wrote:
...
> Perhaps it is a BIOS option ROM like you suggested earlier.  The
> 3c90xC reference manual I found
> (http://people.freebsd.org/~wpaul/3Com/3c90xc.pdf) mentions an option
> ROM (and there is an Atmel part stuffed).  I can't find any technical
> information on the FA331 (yet), so I don't know about it.
> 
> But regardless, wouldn't enumeration have to occur before any option
> ROM could even be used?

Yes. If the device didn't respond with its Device- and Vendor-ID
when reading its configuration space, then it cannot be configured
and reading the expansion ROM won't work.

Thanks,

Anatolij

Re: Failure to detect PCI card

[Wolfgang Denk]

Dear Peter LaDow,

In message <CAN8Q1Ed-YtD1L7e9TaJyELdcz5Rcfdi0mdwU_H6HAx0-3FJ4Lg@mail.gmail.com> you wrote:
> I have a PCI card (a Netgear FA331, vendor:device 100b:0020) that is
> failing to be detected by our PPC platform.  This device works just
> fine in a PC, and other cards work just fine in the same PCI slot (we
> have an Intel 82540EM based card that works).
> 
> But for some reason, neither u-boot nor the kernel detect this card.
> Any ideas why this might be?

Is there any chance that this card has some BIOS (like most graphic
adapters do) which needs to run to put the card into an operative
state (like by initializing any DRAM memory on the card)?  OK, I would
normally not expect such on a 10/100 PCI Ethernet network adapter, but
I don't know this card at all...

Do you know if this card works in any other non-x86 system at all?

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Quote from a recent meeting:   "We are going to continue having these
meetings everyday until I find out why no work is getting done."

Re: Failure to detect PCI card

[Peter LaDow]

On Mon, Aug 5, 2013 at 11:25 AM, Anatolij Gustschin <agust@denx.de> wrote:
> Maybe this card needs bigger delay to respond after PCI reset. You can
> try to re-build U-Boot with defined CONFIG_PCI_BOOTDELAY. Use 1000
> for CONFIG_PCI_BOOTDELAY in the first step and if detection works,
> try to decrease this value.

I'll give this a try.  But a generic question.  Does the kernel depend
upon u-boot to initialize the PCI configuration?

And, I know this is a u-boot specific question, but will u-boot ignore
any PCI device it doesn't know about?  Thus I have to enable a
specific device in the u-boot configuration for u-boot to recognize it
(and from there, have the kernel recognize it)?

On Mon, Aug 5, 2013 at 11:32 AM, Wolfgang Denk <wd@denx.de> wrote:
> Is there any chance that this card has some BIOS (like most graphic
> adapters do) which needs to run to put the card into an operative
> state (like by initializing any DRAM memory on the card)?  OK, I would
> normally not expect such on a 10/100 PCI Ethernet network adapter, but
> I don't know this card at all...

Not that I'm aware of.  I suppose the EEPROM to read in the
configuration/MAC address?  But I wouldn't think an option ROM would
be present.

> Do you know if this card works in any other non-x86 system at all?

Don't have any other non-x86 systems to try out.  We have an
MPC8349E-mITX board, but the PCI slot doesn't detect anything at all
(not even our 82540EM based board).  And this card doesn't work there
either.

Thanks,
Pete