64260/PCI accesses cached by CPU?

64260/PCI accesses cached by CPU?

[Curtis, Allen]

I am using the Artesyn PM/PPC750F PMC in conjunction with a Ramix 100BT quad
Ethernet PMC module. When the system has *low* load, the performance of the
Ethernet ports on the second module is *very* slow. (will not complete `ttcp
-t 10.10.1.1 -s`) If the system is heavily loaded, in this case generating
lots of SCSI disk activity, the same ports have very high performance.

Note:
1. SCSI is also a PCI peripheral
2. The Ethernet ports on the processor are fine

At first I thought this was a hardware, IRQ line, problem but I see the same
behavior on Ethernet ports which do not share interrupt lines. (did not see
anything funny with a scope either)

I have come to the conclusion that the processor is caching either the PCI
I/O or memory access operations. Originally I thought the problem must have
been PCI memory access. I can see from gt64260_find_bridges() the I/O space
is going through ioremap(). However I do not see any such mapping for PCI
memory operations. (nothing in <platform>_setup.c) However after review of
mptbase.c, which works, and pcnet32.c, which is slow, it appears the problem
is reverse.

Linux kernel version:
linux-pmppc750f-prerelease_1_1  (from linux_2_4_galileo)

same results using:
linuxppc_2_4_devel - 2.4.21pre5

Ideas? Suggestions?

Thanks


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: 64260/PCI accesses cached by CPU?

[Mark A. Greer]

Allen,

I just sent essentially the same reply to a private email you sent me
but since you posted here too, I'll also answer here.

If PCI I/O or MEM space was being cached, your PCI drivers wouldn't work
at all...or at least not for very long.  I also mentioned that the
GT64260 is very slow when cache coherency is enabled.  What you
described here sounds like either an interrupt problem--which you've
apparently investigated--or is symptomatic of how the GT64260 handles
cache snooping.  I don't know what your problem is but I think your "PCI
I/O and/or MEM being cached" theory is wrong.  My $0.02.

Also, PCI mem space is mapped by the driver.  That's why you don't see
it mapped by any of the gt64260-specific code.

Mark
--

Curtis, Allen wrote:

>I am using the Artesyn PM/PPC750F PMC in conjunction with a Ramix 100BT quad
>Ethernet PMC module. When the system has *low* load, the performance of the
>Ethernet ports on the second module is *very* slow. (will not complete `ttcp
>-t 10.10.1.1 -s`) If the system is heavily loaded, in this case generating
>lots of SCSI disk activity, the same ports have very high performance.
>
>Note:
>1. SCSI is also a PCI peripheral
>2. The Ethernet ports on the processor are fine
>
>At first I thought this was a hardware, IRQ line, problem but I see the same
>behavior on Ethernet ports which do not share interrupt lines. (did not see
>anything funny with a scope either)
>
>I have come to the conclusion that the processor is caching either the PCI
>I/O or memory access operations. Originally I thought the problem must have
>been PCI memory access. I can see from gt64260_find_bridges() the I/O space
>is going through ioremap(). However I do not see any such mapping for PCI
>memory operations. (nothing in <platform>_setup.c) However after review of
>mptbase.c, which works, and pcnet32.c, which is slow, it appears the problem
>is reverse.
>
>Linux kernel version:
>linux-pmppc750f-prerelease_1_1  (from linux_2_4_galileo)
>
>same results using:
>linuxppc_2_4_devel - 2.4.21pre5
>
>Ideas? Suggestions?
>
>Thanks
>
>
>
>
>
>
>


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: 64260/PCI accesses cached by CPU?

[Allen Curtis]

> If PCI I/O or MEM space was being cached, your PCI drivers wouldn't work
> at all...or at least not for very long.

Only as long as it takes to kick things out of cache. When you are running
md5sum operations of the whole disk drive, this doesn't take very long.
(what was done to generate lots of SCSI I/O) I am sure there are other ways
I could force cache reuse/flushing.

> I also mentioned that the
> GT64260 is very slow when cache coherency is enabled.

Cache coherency is not enabled. It is only enabled if snooping is enabled on
both PCI buses. This isn't the condition with the last Artesyn LSP.

> I don't know what your problem is but I think your "PCI
> I/O and/or MEM being cached" theory is wrong.  My $0.02.

Well, an interesting test would be to run the MPT driver in I/O mode instead
of Memory mode. (the default) There are other interesting things to look at
in the LSP. For instance, the virtual address for the PCI I/O space for each
PCI bus is stored in the "hose" structure. However io.h, uses a constant,
_IO_BASE, for all its in/out() instructions. Each PCI bus I/O space is
ioremap() separately so a single constant offset probably will not work.

> Also, PCI mem space is mapped by the driver.  That's why you don't see
> it mapped by any of the gt64260-specific code.

The MPT driver which uses Memory mapped PCI space works fine. The Ethernet
drivers that use I/O space are not so good.

Hopefully I will have more information next week.


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: 64260/PCI accesses cached by CPU?

[Mark A. Greer]

Allen Curtis wrote:

>>If PCI I/O or MEM space was being cached, your PCI drivers wouldn't work
>>at all...or at least not for very long.
>>
>>
>
>Only as long as it takes to kick things out of cache. When you are running
>md5sum operations of the whole disk drive, this doesn't take very long.
>(what was done to generate lots of SCSI I/O) I am sure there are other ways
>I could force cache reuse/flushing.
>
Not true unless you get very, very lucky.

If a driver is accessing a device/controller's registers through PCI I/O
or MEM space and that space has cacheing enabled, that driver isn't
going to work (unless the cache fills and flushes happen to hit the
registers in "just the right order" (tm).  It won't work because reads
from the controllers registers may return cached data and not really
read the reg at that point in time (consider an interrupt status
register read).  Also, writes to a register will only go into cache and
probably not to the register in the order or at the time required.  It
doesn't matter what else is going on in the system to flush cache.

>>I also mentioned that the
>>GT64260 is very slow when cache coherency is enabled.
>>
>>
>
>Cache coherency is not enabled. It is only enabled if snooping is enabled on
>both PCI buses. This isn't the condition with the last Artesyn LSP.
>
>
>
>>I don't know what your problem is but I think your "PCI
>>I/O and/or MEM being cached" theory is wrong.  My $0.02.
>>
>>
>
>Well, an interesting test would be to run the MPT driver in I/O mode instead
>of Memory mode. (the default) There are other interesting things to look at
>in the LSP. For instance, the virtual address for the PCI I/O space for each
>PCI bus is stored in the "hose" structure. However io.h, uses a constant,
>_IO_BASE, for all its in/out() instructions. Each PCI bus I/O space is
>ioremap() separately so a single constant offset probably will not work.
>
In include/asm-ppc/gt64260.h you will see that the default PCI I/O space
mapping defaults were carefully calculated to be contiguous so both
hoses work with one _IO_BASE_ value.  Its been tested and works.  If
Artysen or you used different values, then all bets are off.  I am
assuming that gt64260_find_bridges() was used.

>>Also, PCI mem space is mapped by the driver.  That's why you don't see
>>it mapped by any of the gt64260-specific code.
>>
>>
>
>The MPT driver which uses Memory mapped PCI space works fine. The Ethernet
>drivers that use I/O space are not so good.
>
>Hopefully I will have more information next week.
>
Okay.

Mark


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: 64260/PCI accesses cached by CPU?

[Johnson, Stephen]

The Artesyn PM/PPC-750F reference kernel is based on the linux_2_4_galileo tree and has not been updated since  _galileo and  _devel merged.

An updated PM/PPC-750F reference kernel is in process...it's in the queue and it's free ($0.00) so it's one of the first things to get bumped down in priority if something comes up that needs attention.




-----Original Message-----
From: Mark A. Greer [mailto:mgreer@mvista.com]
Sent: Monday, May 19, 2003 2:24 PM
To: acurtis@onz.com
Cc: Curtis, Allen; linuxppc-embedded@lists.linuxppc.org
Subject: Re: 64260/PCI accesses cached by CPU?



Allen Curtis wrote:

>>If PCI I/O or MEM space was being cached, your PCI drivers wouldn't work
>>at all...or at least not for very long.
>>
>>
>
>Only as long as it takes to kick things out of cache. When you are running
>md5sum operations of the whole disk drive, this doesn't take very long.
>(what was done to generate lots of SCSI I/O) I am sure there are other ways
>I could force cache reuse/flushing.
>
Not true unless you get very, very lucky.

If a driver is accessing a device/controller's registers through PCI I/O
or MEM space and that space has cacheing enabled, that driver isn't
going to work (unless the cache fills and flushes happen to hit the
registers in "just the right order" (tm).  It won't work because reads
from the controllers registers may return cached data and not really
read the reg at that point in time (consider an interrupt status
register read).  Also, writes to a register will only go into cache and
probably not to the register in the order or at the time required.  It
doesn't matter what else is going on in the system to flush cache.

>>I also mentioned that the
>>GT64260 is very slow when cache coherency is enabled.
>>
>>
>
>Cache coherency is not enabled. It is only enabled if snooping is enabled on
>both PCI buses. This isn't the condition with the last Artesyn LSP.
>
>
>
>>I don't know what your problem is but I think your "PCI
>>I/O and/or MEM being cached" theory is wrong.  My $0.02.
>>
>>
>
>Well, an interesting test would be to run the MPT driver in I/O mode instead
>of Memory mode. (the default) There are other interesting things to look at
>in the LSP. For instance, the virtual address for the PCI I/O space for each
>PCI bus is stored in the "hose" structure. However io.h, uses a constant,
>_IO_BASE, for all its in/out() instructions. Each PCI bus I/O space is
>ioremap() separately so a single constant offset probably will not work.
>
In include/asm-ppc/gt64260.h you will see that the default PCI I/O space
mapping defaults were carefully calculated to be contiguous so both
hoses work with one _IO_BASE_ value.  Its been tested and works.  If
Artysen or you used different values, then all bets are off.  I am
assuming that gt64260_find_bridges() was used.

>>Also, PCI mem space is mapped by the driver.  That's why you don't see
>>it mapped by any of the gt64260-specific code.
>>
>>
>
>The MPT driver which uses Memory mapped PCI space works fine. The Ethernet
>drivers that use I/O space are not so good.
>
>Hopefully I will have more information next week.
>
Okay.

Mark


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/