RE: Yosemite/440EP why are readl()/ioread32() setup to readlittle-endian?

RE: Yosemite/440EP why are readl()/ioread32() setup to readlittle-endian?

[Jenkins, Clive]

A driver for some device that could be connected to (or plugged into)
a variety of different platforms of different architecture.
A driver for a core that could be implemented within an FPGA on
multiple platforms.

Clive

-----Original Message-----
From: Eugene Surovegin [mailto:ebs@ebshome.net]=20
Sent: 02 February 2006 09:08
To: Peter Korsgaard
Cc: Kumar Gala; linuxppc-embedded@ozlabs.org; Jenkins, Clive
Subject: Re: Yosemite/440EP why are readl()/ioread32() setup to
readlittle-endian?


On Thu, Feb 02, 2006 at 09:09:17AM +0100, Peter Korsgaard wrote:
> On 2/2/06, Kumar Gala <galak@kernel.crashing.org> wrote:
> > > What is the preferred way of accessing non-PCI devices then?
Direct
> > > pointer access?
> >
> > No direct pointer access is bad. On PPC You can use
> > in_be{8,16,32}/out_be{8,16,32}
>=20
> What about arch independent drivers? Are there any generic approach
> for this or do you have to stick to ugly #ifdefs to decide between
> in_be32/inl ?

I'm curious, could you give an example of such arch independent=20
driver?

--=20
Eugene

Re: Yosemite/440EP why are readl()/ioread32() setup to readlittle-endian?

[Eugene Surovegin]

On Thu, Feb 02, 2006 at 09:35:56AM -0000, Jenkins, Clive wrote:
> A driver for some device that could be connected to (or plugged into)
> a variety of different platforms of different architecture.
> A driver for a core that could be implemented within an FPGA on
> multiple platforms.

Well, this is all nice but I was wondering about _real_ examples.
When you are talking about "connecting" or "plugging" you have to keep 
in mind actual bus interconnect. So far AFAIK PCI (and derivatives) 
are the only cross-arch bus.

So basically, you have no _real_ life examples, so I'm wondering why 
people need this "arch-independent" non-PCI I/O accessors for 
something which doesn't exist.

I think the reason why Linux doesn't have this stuff follows from the 
previous paragraph.

-- 
Eugene

Re: Yosemite/440EP why are readl()/ioread32() setup to readlittle-endian?

[Matt Porter]

On Thu, Feb 02, 2006 at 01:46:41AM -0800, Eugene Surovegin wrote:
> On Thu, Feb 02, 2006 at 09:35:56AM -0000, Jenkins, Clive wrote:
> > A driver for some device that could be connected to (or plugged into)
> > a variety of different platforms of different architecture.
> > A driver for a core that could be implemented within an FPGA on
> > multiple platforms.
> 
> Well, this is all nice but I was wondering about _real_ examples.
> When you are talking about "connecting" or "plugging" you have to keep 
> in mind actual bus interconnect. So far AFAIK PCI (and derivatives) 
> are the only cross-arch bus.
> 
> So basically, you have no _real_ life examples, so I'm wondering why 
> people need this "arch-independent" non-PCI I/O accessors for 
> something which doesn't exist.
> 
> I think the reason why Linux doesn't have this stuff follows from the 
> previous paragraph.

I mentioned the BE iomap variants that are being used on some non-pci
parisc devices already. I'll give a partial example of something that
is non-pci yet "arch-independent".

Take a non-pci EHCI core (yes, I know it's little endian by definition
but suspend reality for a second).  You can create an arch-independent
EHCI driver that uses the platform bus by using the iomap accessors.
Since these cores are licensed every day by XYZ startups for their
latest "gee-whiz" SoC, it reasons that you'll see the same core on
multiple licensable SoC architectures. I've seen one such thing
on MIPS.

We also know that major semiconductor companies do the same thing
for their peripherals in some cases. They're just as willing to
buy somebody else's USB core, for example.  So, having a BE
non-pci device cross platform isn't a stretch.

Take a look at drivers/scsi/53c700.{c,h}. That generic driver
is why BE iomap accessors were added. It's in the process of
being shared between parisc and m68k.

-Matt

Re: Yosemite/440EP why are readl()/ioread32() setup to readlittle-endian?

[Eugene Surovegin]

On Thu, Feb 02, 2006 at 07:37:01AM -0700, Matt Porter wrote:
> I mentioned the BE iomap variants that are being used on some non-pci
> parisc devices already. I'll give a partial example of something that
> is non-pci yet "arch-independent".
> 
> Take a non-pci EHCI core (yes, I know it's little endian by definition
> but suspend reality for a second).  You can create an arch-independent
> EHCI driver that uses the platform bus by using the iomap accessors.
> Since these cores are licensed every day by XYZ startups for their
> latest "gee-whiz" SoC, it reasons that you'll see the same core on
> multiple licensable SoC architectures. I've seen one such thing
> on MIPS.
> 
> We also know that major semiconductor companies do the same thing
> for their peripherals in some cases. They're just as willing to
> buy somebody else's USB core, for example.  So, having a BE
> non-pci device cross platform isn't a stretch.
> 
> Take a look at drivers/scsi/53c700.{c,h}. That generic driver
> is why BE iomap accessors were added. It's in the process of
> being shared between parisc and m68k.
> 

Matt, my problem with this approach is that it repeats the same 
old mistakes but in "BE-mode", e.g. _assuming_ some access mode and 
hard-coding it into the driver. I fail to see how assuming big-endian 
is any better than assuming little-endian in this case. And this is 
not _portable_ in my book, no matter what some people want me to 
believe.

This fails miserably when for example you have a bus which does byte 
swaps in every half-word. And yes, I have such device on my table and 
I have to port PCMCIA/PCI drivers to this SoC :).

Here is how inb looks like:

static inline u8 fpi_inb(unsigned long port)
{
    port ^= 1;
    return inb(port);
}

IMO, truly portable and driver independent I/O accessors should be 
implemented as a function pointers on per-bus (at least) basis which 
can be overridden by arch or board code. In this case we can get rid 
of "ugly" ifdefs in driver code :).

-- 
Eugene

Re: Yosemite/440EP why are readl()/ioread32() setup to readlittle-endian?

[Matt Porter]

On Thu, Feb 02, 2006 at 09:45:04AM -0800, Eugene Surovegin wrote:
> On Thu, Feb 02, 2006 at 07:37:01AM -0700, Matt Porter wrote:
> > I mentioned the BE iomap variants that are being used on some non-pci
> > parisc devices already. I'll give a partial example of something that
> > is non-pci yet "arch-independent".
> > 
> > Take a non-pci EHCI core (yes, I know it's little endian by definition
> > but suspend reality for a second).  You can create an arch-independent
> > EHCI driver that uses the platform bus by using the iomap accessors.
> > Since these cores are licensed every day by XYZ startups for their
> > latest "gee-whiz" SoC, it reasons that you'll see the same core on
> > multiple licensable SoC architectures. I've seen one such thing
> > on MIPS.
> > 
> > We also know that major semiconductor companies do the same thing
> > for their peripherals in some cases. They're just as willing to
> > buy somebody else's USB core, for example.  So, having a BE
> > non-pci device cross platform isn't a stretch.
> > 
> > Take a look at drivers/scsi/53c700.{c,h}. That generic driver
> > is why BE iomap accessors were added. It's in the process of
> > being shared between parisc and m68k.
> > 
> 
> Matt, my problem with this approach is that it repeats the same 
> old mistakes but in "BE-mode", e.g. _assuming_ some access mode and 
> hard-coding it into the driver. I fail to see how assuming big-endian 
> is any better than assuming little-endian in this case. And this is 
> not _portable_ in my book, no matter what some people want me to 
> believe.
> 
> This fails miserably when for example you have a bus which does byte 
> swaps in every half-word. And yes, I have such device on my table and 
> I have to port PCMCIA/PCI drivers to this SoC :).

Yuck. But a good example that there are always ill-behaved exceptions.

> Here is how inb looks like:
> 
> static inline u8 fpi_inb(unsigned long port)
> {
>     port ^= 1;
>     return inb(port);
> }
> 
> IMO, truly portable and driver independent I/O accessors should be 
> implemented as a function pointers on per-bus (at least) basis which 
> can be overridden by arch or board code. In this case we can get rid 
> of "ugly" ifdefs in driver code :).

There are a ton of reasons for this too, but there's been resistance
in the past to anything adding an additional dereference to the ia32
case.  I think there's been some proposals to get around this and
maybe even some small level of acceptance. However, since the server
folks don't need it, it's slow going to get such a major change pushed
through.

FWIW, some Xscale IXPs could use the per-bus pointer accessors to
manage the some floating I/O windows more cleanly as well. RapidIO
has some use for it too. It's not just byte swapping at least.

You could drive this change, you know. :)

-Matt

Yosemite/440EP PLB4 vs PLB3 DMA to PCI issue

[David Hawkins]

Hi all,

I've been testing the PLB4 and PLB3 DMA controllers
for memory-to-memory transfers between the Yosemite
board SDRAM and a PCI board.

The PCI board contains a PLX PCI-9054 PCI controller
(PCI-to-local bus bridge), and the board contains
SDRAM at the addresses I was testing DMA.

So, the test is basically:

Yosemite SDRAM <-> 440EP bridge <-> PLX-9054 bridge <-> SDRAM

Here's the problem/issue:

PLB4 burst DMA reads are performed as blocks of 32-bytes
(8 transfers of 4-bytes), with the PCI command code
toggling between memory-read-multiple (MRM) and
then memory-read-line (MRL). The change in PCI command
code causes the PLX-9054 controller to consider each
32-byte burst to be a new transaction, and hence
it flushes its internal read FIFO and disconnects from
the local bus for each burst.

The PLB3 DMA controller does not do this, it always
bursts using a memory-read-line (MRL) PCI command code.
Hence the PLX-9054 can determine that the next 32-byte
burst follows on from the previous, and it can deliver
data from its internal FIFOs.

The performance results for transfers between the
Yosemite SDRAM and a device on the 33MHz/32-bit PCI bus were;

   DMA controller    Read          Write
   --------------    ----          -----

      PLB4           19.9MB/s     46.5MB/s

      PLB3           46.0MB/s     46.7MB/s

For the curious, here is the section from my test doc
containing logic analyzer traces;

http://www.ovro.caltech.edu/~dwh/yosemite_440ep_dma.pdf

The performance is below that of the maximum 132MB/s
achievable on the PCI bus. But since the DMA controllers
on the 440EP have to first read and then write,
I don't expect the DMA to do much better than 60MB/s.
So, the ~50MB/s I observed for the majority of tests
seems pretty reasonable.

I haven't had a chance to look at the 440EP PCI bridge
configuration registers that might lead to the use of the
MRM/MRL commands. I'll do that next. I just figured I'd
post these results now, so that others reading this
list might comment (Stefan from Denx comes to mind :))

Cheers
Dave

Re: Yosemite/440EP PLB4 vs PLB3 DMA to PCI issue

[Wolfgang Denk]

In message <43E58202.8020000@ovro.caltech.edu> you wrote:
> 
> MRM/MRL commands. I'll do that next. I just figured I'd
> post these results now, so that others reading this
> list might comment (Stefan from Denx comes to mind :))

He's on vacation; he will be back on Monday, but certainly needs some
time to catch up. Please stay tuned.

Best regards,

Wolfgang Denk

-- 
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Artificial Intelligence is the study of how to  make  real  computers
act like the ones in movies.

Re: Yosemite/440EP PLB4 vs PLB3 DMA to PCI issue

[David Hawkins]

Wolfgang Denk wrote:
> In message <43E58202.8020000@ovro.caltech.edu> you wrote:
> 
>>MRM/MRL commands. I'll do that next. I just figured I'd
>>post these results now, so that others reading this
>>list might comment (Stefan from Denx comes to mind :))
> 
> 
> He's on vacation; he will be back on Monday, but certainly needs some
> time to catch up. Please stay tuned.

No rush, I just wanted to get the question on the list.

I took a look at the 440EP user manual, and I haven't been
able to explain the PLB4 DMA controller observations.

Here's what I've got so far;

p595: PLB-to-PCI transaction handling
    Shows the two situations where the PCI bridge will
    generate memory-read-line (MRL) and memory-read-multiple (MRM)

p646: has comments on PCI Memory to SDRAM DMA Transfer
    and SDRAM to PCI memory DMA transfer. The figures
    for those comments are further on.

    The figures for the PCI read transactions on p663 and p691
    show the PCI command MRL (Eh) being generated on the PCI
    bus, and PCI data bursts occurring in lengths of 32-bytes.

These figures confirm what was observed with the PLB3 DMA
controller. Since the figures show the reads and the writes
on the PLB bus, and do not comment on the PLB bridge,
I can only assume that the figures assume the DMA controller
in use is the PLB3 DMA controller.

As to the PLB4 DMA controller observation of reads to PCI
causing the PCI read command to toggle between MRL and MRM,
I've no idea ... from p595 the transactions are considered
a burst read (MRM), then a burst read of 8-words (MRL),
then repeating. Since each read results in a PCI read burst of
length 32-bytes, there's clearly nothing different between
them on the PCI bus. But perhaps the transaction looks
different on the 440EP internal buses.

The test transfers were to a page-aligned block of SDRAM,
so I don't think the transactions are generating any alignment
issues.

(I hope you had a nice vacation Stefan!)

Cheers
Dave

Re: Yosemite/440EP PLB4 vs PLB3 DMA to PCI issue

[Stefan Roese]

Hi David,

I never used DMA on 440EP/GP till now so I just noticed the two different 
PLB's and their DMA controllers. Never seen this on any 4xx PPC so far. I 
suspect that the PLB4 is integrated mainly because of the USB interface.

On Monday 06 February 2006 00:47, David Hawkins wrote:
> I took a look at the 440EP user manual, and I haven't been
> able to explain the PLB4 DMA controller observations.
>
> Here's what I've got so far;
>
> p595: PLB-to-PCI transaction handling
>     Shows the two situations where the PCI bridge will
>     generate memory-read-line (MRL) and memory-read-multiple (MRM)

What version of the user manual are you referring to? Could you please give 
the pages for the current manual (Revision 1.18).

<snip>

> (I hope you had a nice vacation Stefan!)

Thanks. Very nice. One week of sunshine in the snow. :-)

Ciao,
Stefan

Re: Yosemite/440EP PLB4 vs PLB3 DMA to PCI issue

[David Hawkins]

Hi Stefan,

> I never used DMA on 440EP/GP till now so I just noticed the two different 
> PLB's and their DMA controllers. Never seen this on any 4xx PPC so far. I 
> suspect that the PLB4 is integrated mainly because of the USB interface.

That sounds about right. In my app I won't be using USB, and since the 
PLB4 DMA controller has larger DMA buffers, I thought it might have an
edge over the PLB3 controller.

> What version of the user manual are you referring to? Could you please give 
> the pages for the current manual (Revision 1.18).

Oh, sorry, I was looking at the version 1.14 manual. I've just
downloaded the 1.18 manual. Whoa! The 1.14 manual has 1696 pages,
whereas the 1.18 manual has only 794 - I hope that means things
have got simpler ;)

(nope, looks like the 440 core just got moved into another document)

Anyways, here are the v1.18 page references;

v1.18 p365-6 has the PLB-to-PCI Transaction Handling section
       showing the cases where MRL and MRM will be used.

v1.18 p407 has the PCI Memory to SDRAM DMA transfer section
       with comments and forward references to the timing
       diagram pages.

>>(I hope you had a nice vacation Stefan!)
> 
> Thanks. Very nice. One week of sunshine in the snow. :-)

I hope that means you got some nice snowboarding or skiing!

We have a fun mountain just up the road;
http://www.mammothmountain.com/

Feel free to come and visit if you are in California anytime!

Cheers
Dave

Re: Yosemite/440EP PLB4 vs PLB3 DMA to PCI issue

[Stefan Roese]

Hi David,

On Monday 06 February 2006 20:09, David Hawkins wrote:
> Anyways, here are the v1.18 page references;
>
> v1.18 p365-6 has the PLB-to-PCI Transaction Handling section
>        showing the cases where MRL and MRM will be used.
>
> v1.18 p407 has the PCI Memory to SDRAM DMA transfer section
>        with comments and forward references to the timing
>        diagram pages.

Thanks. I have to admit that my only advice to you is, to send these questions 
to the AMCC support. This seems to be a hardware related question/problem. 
Sorry.

Please keep me informed when you get an explanation for this PLB4 DMA 
behaviour.

> >>(I hope you had a nice vacation Stefan!)
> >
> > Thanks. Very nice. One week of sunshine in the snow. :-)
>
> I hope that means you got some nice snowboarding or skiing!

Yes, skiing! And a little Apres-Ski of course! ;-)

> We have a fun mountain just up the road;
> http://www.mammothmountain.com/
>
> Feel free to come and visit if you are in California anytime!

Thanks for the invitation. But our 9 month's old daughter will not "allow" us 
to make such a long trips in the next few years, I am afraid! :-(

Best regards,
Stefan

Re: Yosemite/440EP PLB4 vs PLB3 DMA to PCI issue

[David Hawkins]

>>Anyways, here are the v1.18 page references;
>>
>>v1.18 p365-6 has the PLB-to-PCI Transaction Handling section
>>       showing the cases where MRL and MRM will be used.
>>
>>v1.18 p407 has the PCI Memory to SDRAM DMA transfer section
>>       with comments and forward references to the timing
>>       diagram pages.
> 
> 
> Thanks. I have to admit that my only advice to you is, to send these questions 
> to the AMCC support. This seems to be a hardware related question/problem. 
> Sorry.

No problem! I appreciate you taking the time to look at it.
I'll contact AMCC and see what they have to say.

> Please keep me informed when you get an explanation for this PLB4 DMA 
> behaviour.

I will.

> Thanks for the invitation. But our 9 month's old daughter will not "allow" us 
> to make such a long trips in the next few years, I am afraid! :-(

Ahh, well come visit in a few years then :)

You can't convince Wolfgang that someone needs to show up at the
Embedded Systems Conference in San Jose, CA, in April, then eh!?

Dave

Re: Yosemite/440EP PLB4 vs PLB3 DMA to PCI issue

[Wolfgang Denk]

Dear David,

in message <43EA3BC9.4090506@ovro.caltech.edu> you wrote:
> 
> You can't convince Wolfgang that someone needs to show up at the
> Embedded Systems Conference in San Jose, CA, in April, then eh!?

Unlikely. To be honest: for me that is not a good  enough  reason  to
get fingerprinted like a criminal. I guess I stay at home.

Ummm... Why don't *you* come to the  Embedded  World  trade  show  in
Nuremberg next week?

Best regards,

Wolfgang Denk

-- 
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
"Don't think; let the machine do it for you!"        - E. C. Berkeley

Re: Yosemite/440EP PLB4 vs PLB3 DMA to PCI issue

[Mark Chambers]

Wolfgang:>
> Unlikely. To be honest: for me that is not a good  enough  reason  to
> get fingerprinted like a criminal. I guess I stay at home.
>

My apologies on behalf of my country.  I've gotten myself on the
excrement list, and I've never even gotten a speeding ticket.  We're
supposed to have this thing called the bill of rights that protects our 
persons
and effects from search without probable cause. Sigh.  Well, maybe
this is to make up for the European fools who gave us ROHS, the
new scourge of hardware designers.  How about if I take my own
chances with terrorists and lead, thank you very much.

Rant of the day,
Mark Chambers

Yosemite/440EP 'issues' as a PCI target

[David Hawkins]

Hi all,

I'm evaluating the 440EP for use as the control processor
on a PCI peripheral. Today I have been looking at what I
can use for processor-to-processor communications over
the PCI bus. The boards will be used in existing
equipment that does not have ethernet in the backplane.

It appears that the 440EP has very limited options for
interrupt-based communications over the PCI bus.

Lets assume the following basic host-to-host protocol
(interrupt-based handshake).

1) PCI system slot machine 'host' has all the IRQs routed
    to it. This will be an x86-based machine.
    The host interrupts the device when it wants to
    send it data, or when it wants to send an acknowledge.

2) PCI device slot; a 440EP based board.
    The device interrupts the host when it wants
    to send it data, or when it wants to send an
    acknowledge.

If the 440EP board wants to interrupt the host, it can
generate an interrupt by writing to the PCI Interrupt
Control/Status Register, PCIC0_ICS, with only 1 bit
available for interrupt generation (i.e., there is
no concept of a mailbox or doorbell register, just a bit)
(see p388 of the v1.18 manual).

Now what if the host wants to interrupt the 440EP.
Errr, there is no register defined for this purpose.
The UIC chapter, p220-222 v1.18 manual indicates
all the interrupt bits. Sure there are a couple of
PCI source interrupts, one for writes to the PCI
configuration-space command register (so can't really
use that), and another for power-management events.

Have I missed something?

I'll have an FPGA/CPLD on the external bus, so I guess
I can implement a mailbox/doorbell register in that
and then have that register trigger an external interrupt
on the 440EP. The 440EP PCI target BARs will then need
to be setup to decode to the EBC decode range.
Sounds like a hack ... (even more of a hack is to
loop back a GPIO onto an EXTINT and setup the target
decode to cover the GPIO registers, and the x86 can
toggle a GPIO directly).

Of course if I have a few unused peripherals I might
be able to cause them to generate an interrupt. But
that gets tricky since in a lot of cases, as device
interrupts are often controlled via DCRs.

I realize that some of the other devices have I2O messaging
units (440GX and 440SP), so are more suited to PCI comms
applications. However, I require the hardware FPU on
the 440EP. From the selector guide, its only the 440EP
and 440EPx that contain the FPU.

Any comments on this?
Dave

Re: Yosemite/440EP 'issues' as a PCI target

[Stefan Roese]

Hi David,

On Friday 10 February 2006 00:58, David Hawkins wrote:
> Now what if the host wants to interrupt the 440EP.
> Errr, there is no register defined for this purpose.
> The UIC chapter, p220-222 v1.18 manual indicates
> all the interrupt bits. Sure there are a couple of
> PCI source interrupts, one for writes to the PCI
> configuration-space command register (so can't really
> use that),

You could use it but I wouldn't recommend it.

> and another for power-management events. 
>
> Have I missed something?

No. This sounds pretty much like the 405GP(r) which also lacks this 
host-to-target interrupt facility.

> I'll have an FPGA/CPLD on the external bus, so I guess
> I can implement a mailbox/doorbell register in that
> and then have that register trigger an external interrupt
> on the 440EP. The 440EP PCI target BARs will then need
> to be setup to decode to the EBC decode range.
> Sounds like a hack ... (even more of a hack is to
> loop back a GPIO onto an EXTINT and setup the target
> decode to cover the GPIO registers, and the x86 can
> toggle a GPIO directly).

Yes, those are the choices available. If the CPU doesn't offer such features 
directly, I wouldn't even call those solutions a "hack". ;-)

> Of course if I have a few unused peripherals I might
> be able to cause them to generate an interrupt. But
> that gets tricky since in a lot of cases, as device
> interrupts are often controlled via DCRs.

I wouldn't go this way.

Best regards,
Stefan

Re: Yosemite/440EP 'issues' as a PCI target

[David Hawkins]

Hi Stefan,

Thanks for confirming my analysis.

There's actually an additional issue with the 440EP
for my application. I'll be using it in a 5V PCI
environment (due to the reuse of the existing host
CPUs). Since the 440EP is not 5V tolerant, I figured
I would add clamps or buffers to the board design.

However, given the meager host-to-host communications
features, I think I would be better off putting an
Intel 21555 non-transparent bridge on the board.
That will provide 5V tolerance, and a full set of
messaging unit and I2O facilities. All for $50-80
or so according to the single-piece pricing from
Digikey. I'm not so happy to need to add another
chip, but if the 440EP passes all the other
benchmark requirements, then it seems the least
painful way to proceed.

Has anyone reading this list had good or bad experiences
with the Intel 21555 or perhaps some of the PLX offers,
eg. PCI 6254?

If only the GX/SP had an FPU ...

Of course there is also the option of finding another
PowerPC that matches my requirements;

   - 300-500MHz CPU
   - ~2W
   - FPU
   - three independent buses; SDRAM, PCI, external
   - the external bus will contain multiple FPGAs
     that generate 128kB of data every 10ms or so.
     The data needs to be DMAed to SDRAM, where
     the host CPU can convert it to float, FFT
     it, process, and average the data. Transfers
     to host over PCI occur every 100ms.

     FPGA-to-SDRAM should occur in ~1ms;
        128kB/1ms = 128MB/s

     There will be up to 20 boards in a crate,
     and transfers from all 20 boards need to
     complete in 100ms, so

     FPGA-to-host should occur in ~5ms;
       128kB/5ms = 25MB/s

     So I don't need stunning PCI performance, but
     I do need a reasonable external memory bus
     bandwidth.

     The 440EP 16bit/66MHz 132MB/s would just meet
     my requirement (and I can handle a 50% drop
     in bus bandwidth if benchmarks go that way).
     The PCI performance hits 50MB/s, so its ok.

     I don't want to use a local-bus PCI interface
     with the FPGAs, since then I'd need a PCI core
     in each. I typically pack the FPGAs to 90%
     with processing logic, so can't afford the
     space for a complex host-to-FPGA interface.

I think I've shown you the current boards (that
use a TI DSP);

http://www.ovro.caltech.edu/~dwh/correlator

I had looked at the MPC8245 processor a while back,
but its SDRAM interface is multiplexed with its external
memory bus, so DMA from the external bus to SDRAM
would likely be pretty poor.

Do you have any experience with features of the PowerQUICC
processors? I've tried to avoid a full-up G4/G5 processor
since they typically also require a system controller
chip, and consume a lot more power.

I had also considered using a ColdFire processor, but
went with the PowerPC since I'll be using some Virtex
FPGAs with the PowerPC in a future project.

Anyway, just thought I'd give you an idea of what I'm
trying to figure out.

Cheers,
Dave

Re: Yosemite/440EP 'issues' as a PCI target

[Andrew Armitage]

Hi David,

> However, given the meager host-to-host communications
> features, I think I would be better off putting an
> Intel 21555 non-transparent bridge on the board.
> That will provide 5V tolerance, and a full set of
> messaging unit and I2O facilities. All for $50-80
> or so according to the single-piece pricing from
> Digikey. I'm not so happy to need to add another
> chip, but if the 440EP passes all the other
> benchmark requirements, then it seems the least
> painful way to proceed.
> 
> Has anyone reading this list had good or bad experiences
> with the Intel 21555 or perhaps some of the PLX offers,
> eg. PCI 6254?

We've used the 21555 on a couple of our designs (mated to a 7457/107 
pair) and have never had too many problems with them.  The messaging 
unit has been very helpful.

Just my $0.02,

Andrew

Re: Yosemite/440EP 'issues' as a PCI target

[David Hawkins]

> We've used the 21555 on a couple of our designs (mated to a 7457/107 
> pair) and have never had too many problems with them.  The messaging 
> unit has been very helpful.
> 
> Just my $0.02,

Hey Andrew,
Thanks!
Dave

Re: Yosemite/440EP 'issues' as a PCI target

[Wolfgang Denk]

Dear David,

in message <43ECC7CE.1010409@ovro.caltech.edu> you wrote:
> 
> Of course there is also the option of finding another
> PowerPC that matches my requirements;
...

MPC834x comes to mind. But I have to admit that we never tried  using
this as a PCI device yet...

Best regards,

Wolfgang Denk

-- 
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
I can type faster than I can move a  mouse,  so  I  find  menu-driven
drawing packages time consuming and frustrating.      - W. R. Stevens

Re: Yosemite/440EP 'issues' as a PCI target

[David Hawkins]

>>Of course there is also the option of finding another
>>PowerPC that matches my requirements;
> 
> ...
> 
> MPC834x comes to mind. But I have to admit that we never tried  using
> this as a PCI device yet...
> 

Thanks Wolfgang, I'll take a look at the user manual.

Regarding PowerPC devices that can be (uselessly) configured
as PCI agents/targets/peripherals;

  - the Artesyn PMC board I got from eBay contains the
    IBM CPC700 bridge - it can never generate PCI interrupts
    as a target (non-monarch mode).

  - the MPC5200 PowerPC, can also be configured as a target,
    but will never generate PCI interrupts.

Gee, who knew. I didn't :)

Dave

Re: Yosemite/440EP 'issues' as a PCI target

[David Hawkins]

Hey Travis,

 >> Not great, so any suggestions for whats better?
 >>
 > Everyone uses them because they're easy and just work. Sometimes
 > you may need to tweak the setup eeprom, but in general they just work.
 > We use a pericom 7300 (can't remember and working from home
 > today), that is quite nice.  Not an issue with it at all. HW
 > guy just plunked it down and it worked.  They have an eval board to.

I was going to take a look around TIs site, I'd forgotten Pericom
makes bridges too, I'll read the data book.

 > ... I've not had a need to use it, my work is 90% PCI
 > (forwarding engine, queueing engine, etc) for 10G ethernet, so
 > FPU stuff is very rare, and not time critical (used for setting
 > up policing/rate limiting etc). Basically, set it and forget it
 > calculations.

I figured that was probably the case - and the reason the majority
of the embedded processors are sans-FPU. I didn't want to use a
TI DSP again, there's just no 'community spirit' like with Linux.

So, I'll take a look at the PowerQUICC that Wolfgang mentioned, and
see if I want to change to that processor, another look at the ColdFire
(I have an MCF5485 eval board), and a look at the bridge data sheets
and see where that leaves me.

Personally, I want to stay with the PowerPC, the developer mailing
list is great!

Thanks,
Dave

Re: Yosemite/440EP 'issues' as a PCI target

[David Hawkins]

Hi Wolfgang,

>>Of course there is also the option of finding another
>>PowerPC that matches my requirements;
> ...
> MPC834x comes to mind. But I have to admit that we never tried  using
> this as a PCI device yet...

Indeed you were correct. The MPC834x series meets my requirements.
It is also a 3.3V PCI device, so I'm checking into whether
the Trenton CPUs I can use as host CPUs use a 3.3V PCI interface.
If that is the case, then I'll move the Force CPUs into another
system, and I'll just define the cPCI bus in the correlator
system as 3.3V-only.

The potential advantages of the MPC834x over the 440EP are;

   - it has doorbell registers and mailboxes for PCI
     host-to-host comms (though does not have an I2O interface)

   - its DDR-SDRAM controller can run faster than the 440EP

   - its external local bus is wider and faster than
     the 440EP

   - but I think the MPC834x internal buses are slower than
     the 440EP CoreConnect buses, so I'll need to benchmark
     to compare the two.

   - it exists! I can find them on distributors web sites
     (can't say the same for the 440EP yet!)

So, I'll continue with my evalation of the 440EP, and then
I'll want to evaluate the MPC834x devices.

What MPC834x development kit are you using? What would you recommend?

  - I'd like to get a board in a PCI or cPCI-form factor, since
    I could use it as a PCI agent/peripheral
  - I'd really like to get access to the external bus
    too.

Embedded Planet has a PrPMC module with an 8343E, but I doubt
I can get to the local bus on that board. I found news releases
of an MPC8349E QUICCstart board, but couldn't find anything
on Freescale's web site. Wind River appears to have an eval
board; SBC8347E/49E, which looks to be a cPCI board, but their
web site didn't have much in the way of details (or pictures).

Freescale do have the MPC8349EMDS 'MPC8349E Modular Development
System', but its not clear how much of that system I would
need to purchase just to test the board in a PCI system.
Looks like its aimed toward AdvancedTCA development.

Anyway, thanks for the pointer to the MPC834x series!

Dave

Re: Yosemite/440EP 'issues' as a PCI target

[Stefan Roese]

Hi David,

On Saturday 11 February 2006 07:03, David Hawkins wrote:
> Indeed you were correct. The MPC834x series meets my requirements.
> It is also a 3.3V PCI device, so I'm checking into whether
> the Trenton CPUs I can use as host CPUs use a 3.3V PCI interface.
> If that is the case, then I'll move the Force CPUs into another
> system, and I'll just define the cPCI bus in the correlator
> system as 3.3V-only.

Ahhh. ;-)

> The potential advantages of the MPC834x over the 440EP are;
>
>    - it has doorbell registers and mailboxes for PCI
>      host-to-host comms (though does not have an I2O interface)
>
>    - its DDR-SDRAM controller can run faster than the 440EP
>
>    - its external local bus is wider and faster than
>      the 440EP
>
>    - but I think the MPC834x internal buses are slower than
>      the 440EP CoreConnect buses, so I'll need to benchmark
>      to compare the two.
>
>    - it exists! I can find them on distributors web sites
>      (can't say the same for the 440EP yet!)

Hmmm. The 440EP is around for quite a while. I would be surprised if you 
couldn't buy those parts right now. But I have to admit, I never tried to. I 
would contact my AMCC distributor for availability...

Best regards,
Stefan

Re: Yosemite/440EP 'issues' as a PCI target

[David Hawkins]

> Hmmm. The 440EP is around for quite a while. I would be surprised if you 
> couldn't buy those parts right now. But I have to admit, I never tried to. I 
> would contact my AMCC distributor for availability...

I always figure that if you can find it on a distributor web
site without having to explicitly ask for the part, then
it really is available. I got my Yosemite kit through
Avnet, so lets start there:

1. Avnet

    http://www.em.avnet.com

    Search for 440EP from the vendor AMCC, and it only shows up
    the eval boards.

    Search for 'contains' MPC834 from Freescale, and there are
    lots of hits (though no stock for most of them).

2. Arrow North America

    http://www.arrownac.com

    No AMCC stuff there.

    A search for the Freescale part shows up a bunch, although they
    have no stock either.

The successful search hits for the Freescale part gives me
more confidence in those parts. However, you are correct, if I
really want to know the availablity of parts, I should talk
directly to an AMCC rep.

Cheers
Dave

Re: Yosemite/440EP 'issues' as a PCI target

[Wolfgang Denk]

In message <43ED7E1E.2030700@ovro.caltech.edu> you wrote:
> 
> What MPC834x development kit are you using? What would you recommend?

We use (*) the TQM834x modules (with a STK83xx/85xx starter kit).

(*) "use" means that we ported U-Boot and Linux for it.

>   - I'd like to get a board in a PCI or cPCI-form factor, since
>     I could use it as a PCI agent/peripheral

Both PCI and cPSI are present on the STK83xx board.

>   - I'd really like to get access to the external bus
>     too.

Available on headers, too

Feel free to contact me for details.


Best regards,

Wolfgang Denk

-- 
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
To know how another being, another creature feels -  that  is  impos-
sible.                  - Terry Pratchett, _The Dark Side of the Sun_

Re: Yosemite/440EP 'issues' as a PCI target

[Stefan Roese]

Hi David,

On Friday 10 February 2006 18:05, David Hawkins wrote:
> There's actually an additional issue with the 440EP
> for my application. I'll be using it in a 5V PCI
> environment (due to the reuse of the existing host
> CPUs).

Autsch! Those must be pretty old CPU's! Is this standard desktop PCI or 
CompactPCI or PMC?

> Since the 440EP is not 5V tolerant, I figured 
> I would add clamps or buffers to the board design.

I would be careful here, since you easily can violate the pci specs. Do you 
have other pci devices on this pci bus?

Just use newer host CPU's! ;-)

Best regards,
Stefan

Re: Yosemite/440EP 'issues' as a PCI target

[David Hawkins]

>>There's actually an additional issue with the 440EP
>>for my application. I'll be using it in a 5V PCI
>>environment (due to the reuse of the existing host
>>CPUs).
> 
> Autsch! Those must be pretty old CPU's! Is this standard desktop PCI or 
> CompactPCI or PMC?

They're not too old ... but at $2k a piece, you don't want
to just throw them away.

The CPUs are Force Computers 735AR2 devices. They contain
a Force Sentinel PCI-to-PCI bridge. The bridges can operate
at 3.3V or 5V, but they decided to hardwire them for 5V
operation to implement hot-swap.

The other newer CPUs are Trenton CPLEs.
http://www.trentontechnology.com/products/singleboards/5932.php4

The system uses cPCI crates. In some cases 21152 bridges are
used to make larger backplanes, in other cases SBS bridges
containing 21152's are used to link crates.

>>Since the 440EP is not 5V tolerant, I figured 
>>I would add clamps or buffers to the board design.
> 
> I would be careful here, since you easily can violate the pci
> specs. Do you  have other pci devices on this pci bus?

Yeah, I'd make sure that I had others check my work. However,
the 21555 would take care of it, and the only PCI bus
on the board would be the one between the 440EP and the
bridge.

> Just use newer host CPU's! ;-)

Ah, the words of a true engineer :)

Of course the project manager wouldn't appreciate me doing that.

However, its not totally out of the question ... $50 per 21555
and 20 per crate, thats $1000, about half the price of
a CPU. But of course, without the I20 unit on the 440EP,
I might need the 21555 anyway.

Cheers
Dave

Re: Yosemite/440EP 'issues' as a PCI target

[Stefan Roese]

Hi David,

On Friday 10 February 2006 19:11, David Hawkins wrote:
> > Just use newer host CPU's! ;-)
>
> Ah, the words of a true engineer :)
>
> Of course the project manager wouldn't appreciate me doing that.
>
> However, its not totally out of the question ... $50 per 21555
> and 20 per crate, thats $1000, about half the price of
> a CPU. But of course, without the I20 unit on the 440EP,
> I might need the 21555 anyway.

Do you really need this I2O unit? You could easily create some message 
ringbuffers, one in the 440EP's SDRAM for the host-to-440 messages and one in 
the host-cpu SDRAM for the 440-to-host messages. This way, all messages will 
be transferred using pci writes.

Best regards,
Stefan

Re: Yosemite/440EP 'issues' as a PCI target

[David Hawkins]

>>However, its not totally out of the question ... $50 per 21555
>>and 20 per crate, thats $1000, about half the price of
>>a CPU. But of course, without the I20 unit on the 440EP,
>>I might need the 21555 anyway.
> 
> 
> Do you really need this I2O unit? You could easily create some message 
> ringbuffers, one in the 440EP's SDRAM for the host-to-440 messages and one in 
> the host-cpu SDRAM for the 440-to-host messages. This way, all messages will 
> be transferred using pci writes.

Hi Stefan,

Nope, I can always work with the 440EP. I was just disappointed
to find that the 440EP had no way for an external host to
generate an interrupt (without coming up with the work-arounds
I suggested). I don't have the part designed into a board, I'm
merely  evaluating it. So, I wanted to see what else was out
there with a similar set of features; the MPC834x has them.

So, now I have two possible processors to evaluate :)

I'll finish the other tests I want to do with the 440EP, get
myself a MPC834x and repeat the same tests with it (which
shouldn't take too long, since it'll be a repeat of the
440 tests).

The MPC834x does have features to make communications between a
host and a target (PCI 'agent') work much nicer than the 440EP.
It also appears to have a faster local bus. I haven't had time
to benchmark, or estimate from timing diagrams, the performance
of the 440EP external memory bus, but there is a chance that
it will fail to meet my spec.

I wouldn't consider the 440EP work to be wasted. Its been a good
learning experience, and mainly I've just been working on this
stuff in the evenings and weekends to see whether the processor
will work. This was my first experience with PowerPC processors,
so its all good fun!

Cheers
Dave

Re: Yosemite/440EP 'issues' as a PCI target

[David Hawkins]

> Do you really need this I2O unit? You could easily create some message 
> ringbuffers, one in the 440EP's SDRAM for the host-to-440 messages and one in 
> the host-cpu SDRAM for the 440-to-host messages. This way, all messages will 
> be transferred using pci writes.

Actually, its not the I2O unit I want, really just the doorbell
registers in the messaging unit. The MPC834x's seem to have
dropped the I2O units in that the 8272-family appears to
have, but they have the messaging unit registers.

I plan to create something like the ring-buffers scheme you
mention. And of course, you don't need an I2O unit to
implement it. However, you do need to be able to interrupt
the 440 from the host - which was the original issue.
But, there is a work-around :)

BTW, even with this scheme you really want to use DMA on
the adapter board for transfers in either direction, i.e.,
use the peripheral for both DMA read and write.

Unless of course you want to turn your 1GHz x86 host processor
into a 1MHz CPU. Check out the following performance tests
that I did between the current boards and an x86 host:

http://www.ovro.caltech.edu/~dwh/correlator/pdf/pci_performance.pdf

The boards use a PLX-9054 bridge and can DMA between themselves
at 118MB/s. However, the DMA through bridges, and DMA to/from host
bandwidth is less due to burst capabilites (or lack thereof) of
the bridges. CPU reads/writes by the host were dismal; look
at p19-26.

Of course, if I had used a PowerPC host I'm sure I would
have been fine!

Dave

Re: Yosemite/440EP 'issues' as a PCI target

[David Hawkins]

Hi Mark,

> Some unsolicited brainstorming

Brainstorm away!

> if you go with a QUICC based processor you could bring 
> the data into SDRAM via a serial port. That gets you DMA
> and some extra buffer management help from the co-processor.

Ooooh I like it.

> Some extra logic in the FPGA (but less pins), but you've got
> a lot of flexibility in what you can do with SCCs or I2C.

But do I have the bandwidth? (I'll look in the data book).

> I'm currently working on an MPC8247 based design, btw.
> It has a PC104+ header and is directly connected to a 
> T.I. 6205 (which, at $10, is a lot of crunch for the buck).

But the you have to program it with Code Composer and
the horrid TI DSP tools. Yeah, ok I've got the older
C31 version, not the newer 6x-based DSPs. I'm trying
to move the system to Linux, so that others round
here can 'share the burden'. I'm designing the boards,
drivers, and host software (ACE/C++) at the moment.
I need a break! :)

I want to avoid spending $20k on the TI tools and an
emulator ... hey, call me cheap.

However, I like the SCC idea. They sound very similar
to the multi-processor links on the Analog Devices
SHARC DSPs.

More reading :)

Dave

Re: Yosemite/440EP 'issues' as a PCI target

[David Hawkins]

Hi Mark,

> I'm currently working on an MPC8247 based design, btw.
> It has a PC104+ header and is directly connected to a
> T.I. 6205 (which, at $10, is a lot of crunch for the buck).

I took a look at the MPC8727 family.

The CPM runs at 200MHz max, and the SSCs can clock at
1/4 that, so only 50MHz; 50MHz/8-bits = 6MB/s.
They're too slow to move 132MB/s.

There is only two external buses too, and no
external bus to memory DMA, since they are the
same bus.

Also, the MPC834x series exists, and beats it out
performance wise.

Thanks for the pointer though!

Dave

Re: Yosemite/440EP 'issues' as a PCI target

[David Hawkins]

Hi Travis,

>>> Ah, the ol' dec 21555, everyone has had to use one in their lifetime...
>>> Not bad, not great...
>>
>> Not great, so any suggestions for whats better?
>>
> Everytone uses them because they're easy and just work. Sometimes you 
> may need to tweak the setup eeprom, but in general they just work.
> We use a pericom 7300 (can't remember and working from home today), that
> is quite nice.  Not an issue with it at all. HW guy just plunked it down 
> and it worked.  They have an eval board to.

The 7300 is a PCI-to-PCI bridge that only operates in transparent
mode, it does not have any I2O capabilities. I used the Pericom
cross-reference guide and they don't have a replacement part
for the 21555, or the PLX6254 or PLX6540. So, thanks for the
suggestion, but the 21555 still has it.

I didn't find any Pericom bridge that was non-transparent.

Dave

RE: Yosemite/440EP why are readl()/ioread32() setup to readlittle-endian?

[Jenkins, Clive]

>>>p453 of his book has a PCI DMA example, where he uses the
>>>cpu_to_le32() macros inside calls writel().

>>>However, since these functions are internally implemented to
>>>perform LE operations, this example appears to be incorrect.

>>>Would you agree?
=20
>> No, I think it is correct.

I said this without looking up the example you cited.
I agree now, the example is incorrect; and yes, file a bug
report at oreilly!

I think, from memory, that elsewhere in the book Rubini does
say that readl()... are for the PCI bus, but cross-arch issues
are only addressed in certain sections.

I always find out exactly what these macros do on the arch
I am using, then I know where I stand. I find LXR (Google it if
you don't know it) good for browsing the source of vanilla
kernels. After finding out how and where it is done, I then
double check the relevant files of the actual kernel I am using.

ppc implementations of readl, writel, cpu_to_le32 use the byte-
reversed load/store word instructions.

Clive

Re: Yosemite/440EP why are readl()/ioread32() setup to readlittle-endian?

[David Hawkins]

> I said this without looking up the example you cited.
> I agree now, the example is incorrect; and yes, file a bug
> report at oreilly!

Ok, I will.

> I think, from memory, that elsewhere in the book Rubini does
> say that readl()... are for the PCI bus, but cross-arch issues
> are only addressed in certain sections.

I didn't find anything that specifically mentioned their
use was for the PCI bus only. The endianness swapping
features of the pci_config_xxx functions are clearly
stated, but not the readl/writel. And of course the
example I refer to clearly uses those functions on the
PCI bus incorrectly.

But its still a great book.

> I always find out exactly what these macros do on the arch
> I am using, then I know where I stand. I find LXR (Google it if
> you don't know it) good for browsing the source of vanilla
> kernels. After finding out how and where it is done, I then
> double check the relevant files of the actual kernel I am using.
> 
> ppc implementations of readl, writel, cpu_to_le32 use the byte-
> reversed load/store word instructions.

Ahh, very good advice. I think I read about LXR in one of
Freescale's app notes on porting Linux. I'll go take a look
on Google.

Thanks
Dave

RE: Yosemite/440EP why are readl()/ioread32() setup to readlittle-endian?

[Jenkins, Clive]

> >>readl() and ioread32() read the registers in little-endian format!
> >
> > Correct. That's how it is implemented on all platforms. Think for
> > example of an pci device driver. Using these IO functions, the
> > driver will become platform independent, running without
> > modifications on little- and big-endian machines.
>
> I just stumbled across the section in Rubini 3rd Ed that mislead
> me into believing that the readl()/writel() were machine endianness
> dependent, i.e., LE on x86, BE on PPC.

> p453 of his book has a PCI DMA example, where he uses the
> cpu_to_le32() macros inside calls writel().

> However, since these functions are internally implemented to
> perform LE operations, this example appears to be incorrect.

> Would you agree?

No, I think it is correct.

On most architectures readl() and writel() assume you are
accessing MMIO on the PCI bus, which is little-endian.
So these macros convert between the endian-ness of the CPU
and the endian-ness of the PCI bus.

Clive

Re: Yosemite/440EP why are readl()/ioread32() setup to readlittle-endian?

[David Hawkins]

Jenkins, Clive wrote:
>>>>readl() and ioread32() read the registers in little-endian format!
>>>
>>>Correct. That's how it is implemented on all platforms. Think for
>>>example of an pci device driver. Using these IO functions, the
>>>driver will become platform independent, running without
>>>modifications on little- and big-endian machines.
>>
>>I just stumbled across the section in Rubini 3rd Ed that mislead
>>me into believing that the readl()/writel() were machine endianness
>>dependent, i.e., LE on x86, BE on PPC.
> 
> 
>>p453 of his book has a PCI DMA example, where he uses the
>>cpu_to_le32() macros inside calls writel().
> 
> 
>>However, since these functions are internally implemented to
>>perform LE operations, this example appears to be incorrect.
> 
> 
>>Would you agree?
> 
> 
> No, I think it is correct.
> 
> On most architectures readl() and writel() assume you are
> accessing MMIO on the PCI bus, which is little-endian.
> So these macros convert between the endian-ness of the CPU
> and the endian-ness of the PCI bus.
> 
> Clive

Hey Clive,

Right, but in the 440EP source, and probably on other PowerPC
ports, readl() and writel() perform the endian conversion for
you, so if you wrote an operation as

   writel(register_address, cpu_to_le32(data));

you would get *two* little endian conversions, i.e., you
would write the bytes in the wrong order.

I haven't looked in the source for other big-endian architecures
yet. I wonder if the ARM stuff is big, or little endian?

The original question came about while I was trying to read
back PLX-9054 registers (a PCI-to-local bus bridge on a PCI
adapter board).

I search through the other drivers for the 440EP peripherals
that are not on the PCI bus showed that the authors used
the in_be32() and out_be32() calls. The comment at the top
of this email indicates that the readl() and writel() are
effectively 'reserved' for PCI accesses. I didn't get this
impression from reading Rubini.

Thanks for you comments. Feel free to add more comments!

Cheers
Dave

Re: Yosemite/440EP why are readl()/ioread32() setup to readlittle-endian?

[Matt Porter]

On Wed, Feb 01, 2006 at 09:02:31AM -0800, David Hawkins wrote:
> Jenkins, Clive wrote:
> >>>>readl() and ioread32() read the registers in little-endian format!
> >>>
> >>>Correct. That's how it is implemented on all platforms. Think for
> >>>example of an pci device driver. Using these IO functions, the
> >>>driver will become platform independent, running without
> >>>modifications on little- and big-endian machines.
> >>
> >>I just stumbled across the section in Rubini 3rd Ed that mislead
> >>me into believing that the readl()/writel() were machine endianness
> >>dependent, i.e., LE on x86, BE on PPC.
> > 
> > 
> >>p453 of his book has a PCI DMA example, where he uses the
> >>cpu_to_le32() macros inside calls writel().
> > 
> > 
> >>However, since these functions are internally implemented to
> >>perform LE operations, this example appears to be incorrect.
> > 
> > 
> >>Would you agree?
> > 
> > 
> > No, I think it is correct.
> > 
> > On most architectures readl() and writel() assume you are
> > accessing MMIO on the PCI bus, which is little-endian.
> > So these macros convert between the endian-ness of the CPU
> > and the endian-ness of the PCI bus.
> > 
> > Clive
> 
> Hey Clive,
> 
> Right, but in the 440EP source, and probably on other PowerPC
> ports, readl() and writel() perform the endian conversion for
> you, so if you wrote an operation as
> 
>    writel(register_address, cpu_to_le32(data));
> 
> you would get *two* little endian conversions, i.e., you
> would write the bytes in the wrong order.
> 
> I haven't looked in the source for other big-endian architecures
> yet. I wonder if the ARM stuff is big, or little endian?

Both

> The original question came about while I was trying to read
> back PLX-9054 registers (a PCI-to-local bus bridge on a PCI
> adapter board).

Those regs are little endian since it's a PCI device. You
use the old read*/write* or new ioread*/iowrite* since it's
a PCI device.

> I search through the other drivers for the 440EP peripherals
> that are not on the PCI bus showed that the authors used
> the in_be32() and out_be32() calls. The comment at the top
> of this email indicates that the readl() and writel() are
> effectively 'reserved' for PCI accesses. I didn't get this
> impression from reading Rubini.

The book implicitly focuses on x86 driver developers, that's
why you don't get an explicit statement about this...
"everything" is PCI in that world.

read*/write* and ioread*/iowrite* generate outbound little
endian cycles on ALL arches, period.  They are intended
only for PCI use and have generic names only because of
the assumption that "all the world is a PC".

Now, what it takes to to generate outbound little endian cycles
varies. On some arches, it's just a store (native LE) on
other arches, it's a reversed store (PPC), others still configure
their PCI bridge hardware to do byte swapping in hardware (typically
if their arch doesn't have a simple byte-swapping store like PPC).

The example you cite on pg. 453 of Rubini looks broken for BE
systems. It works on LE systems since cpu_to_le32() does nothing
and writel is a simply dereference. That's pure luck. On PPC,
for example, that would write a big endian bus_addr to the fictitious
PCI device which is not what they want.

-Matt

Re: Yosemite/440EP why are readl()/ioread32() setup to readlittle-endian?

[David Hawkins]

> The book implicitly focuses on x86 driver developers, that's
> why you don't get an explicit statement about this...
> "everything" is PCI in that world.
> 
> read*/write* and ioread*/iowrite* generate outbound little
> endian cycles on ALL arches, period.  They are intended
> only for PCI use and have generic names only because of
> the assumption that "all the world is a PC".
> 
> Now, what it takes to to generate outbound little endian cycles
> varies. On some arches, it's just a store (native LE) on
> other arches, it's a reversed store (PPC), others still configure
> their PCI bridge hardware to do byte swapping in hardware (typically
> if their arch doesn't have a simple byte-swapping store like PPC).
> 
> The example you cite on pg. 453 of Rubini looks broken for BE
> systems. It works on LE systems since cpu_to_le32() does nothing
> and writel is a simply dereference. That's pure luck. On PPC,
> for example, that would write a big endian bus_addr to the fictitious
> PCI device which is not what they want.

Great! An authoritive answer!

Re: endianness, even cooler on the 440EP, in your mmap()
implementation you can set the _PAGE_ENDIAN flag, and
user-space will see the PCI device in little endian
format. Fun stuff!

Thanks Matt.

Dave

Re: Yosemite/440EP why are readl()/ioread32() setup to readlittle-endian?

[David Hawkins]

Matt,

In the same vein as the readl()/writel() question, what
are the assumptions regarding memcpy_toio and memcpy_fromio?

If the memcpy_to/fromio operations are intended only
for access to PCI devices, then they should also inherently
perform little-endianness conversion. For the test driver
I was working on, I did *not* find this the case, eg.
I implemented the test driver read() and write() using the
memcpy_to/fromio calls, and the data transfers occur
in big-endian (well, 'native' mode, since I also test the
same test driver with the PCI adapter in an x86 system).

If memcpy_to/fromio can be used in a more general context,
then I can see why they operate in native mode.

Just looking for enlightenment.

Cheers
Dave

Re: Yosemite/440EP why are readl()/ioread32() setup to readlittle-endian?

[Eugene Surovegin]

On Wed, Feb 01, 2006 at 10:04:15AM -0800, David Hawkins wrote:
> Matt,
> 
> In the same vein as the readl()/writel() question, what
> are the assumptions regarding memcpy_toio and memcpy_fromio?
> 
> If the memcpy_to/fromio operations are intended only
> for access to PCI devices, then they should also inherently
> perform little-endianness conversion. For the test driver
> I was working on, I did *not* find this the case, eg.
> I implemented the test driver read() and write() using the
> memcpy_to/fromio calls, and the data transfers occur
> in big-endian (well, 'native' mode, since I also test the
> same test driver with the PCI adapter in an x86 system).
> 
> If memcpy_to/fromio can be used in a more general context,
> then I can see why they operate in native mode.
> 
> Just looking for enlightenment.

This commands IIRC are intended for copying chunk of _bytes_. There 
are no issues with endianess for bytes, e.g. they work just like 
ordinary memcpy.

-- 
Eugene

Re: Yosemite/440EP why are readl()/ioread32() setup to readlittle-endian?

[David Hawkins]

Eugene Surovegin wrote:
> On Wed, Feb 01, 2006 at 10:04:15AM -0800, David Hawkins wrote:
> 
>>Matt,
>>
>>In the same vein as the readl()/writel() question, what
>>are the assumptions regarding memcpy_toio and memcpy_fromio?
>>
>>If the memcpy_to/fromio operations are intended only
>>for access to PCI devices, then they should also inherently
>>perform little-endianness conversion. For the test driver
>>I was working on, I did *not* find this the case, eg.
>>I implemented the test driver read() and write() using the
>>memcpy_to/fromio calls, and the data transfers occur
>>in big-endian (well, 'native' mode, since I also test the
>>same test driver with the PCI adapter in an x86 system).
>>
>>If memcpy_to/fromio can be used in a more general context,
>>then I can see why they operate in native mode.
>>
>>Just looking for enlightenment.
> 
> 
> This commands IIRC are intended for copying chunk of _bytes_. There 
> are no issues with endianess for bytes, e.g. they work just like 
> ordinary memcpy.
> 

True, good point.

I quite often implement a 'control' device to read/write/mmap PCI
device registers. In that case, the registers are usually 32-bit, so
if I wanted endian neutrality, I could either let the user-space
app determine the endianness and act accordingly, or force the
user-space app to always see little-endian registers by replacing
memcpy_to/fromio calls with a loop over read;/writel, and in mmap
making sure to set the _PAGE_ENDIAN flag. Of course, making mmap
endian-neutral depends on the 440EP page flags, which say an
ARM might not have.

Thanks for the valuable feedback guys.

Dave

Re: Yosemite/440EP why are readl()/ioread32() setup to readlittle-endian?

[Eugene Surovegin]

On Wed, Feb 01, 2006 at 10:20:42AM -0800, David Hawkins wrote:
> Eugene Surovegin wrote:
> >On Wed, Feb 01, 2006 at 10:04:15AM -0800, David Hawkins wrote:
> >
> >>Matt,
> >>
> >>In the same vein as the readl()/writel() question, what
> >>are the assumptions regarding memcpy_toio and memcpy_fromio?
> >>
> >>If the memcpy_to/fromio operations are intended only
> >>for access to PCI devices, then they should also inherently
> >>perform little-endianness conversion. For the test driver
> >>I was working on, I did *not* find this the case, eg.
> >>I implemented the test driver read() and write() using the
> >>memcpy_to/fromio calls, and the data transfers occur
> >>in big-endian (well, 'native' mode, since I also test the
> >>same test driver with the PCI adapter in an x86 system).
> >>
> >>If memcpy_to/fromio can be used in a more general context,
> >>then I can see why they operate in native mode.
> >>
> >>Just looking for enlightenment.
> >
> >
> >This commands IIRC are intended for copying chunk of _bytes_. There 
> >are no issues with endianess for bytes, e.g. they work just like 
> >ordinary memcpy.
> >
> 
> True, good point.
> 
> I quite often implement a 'control' device to read/write/mmap PCI
> device registers. In that case, the registers are usually 32-bit, so
> if I wanted endian neutrality, I could either let the user-space
> app determine the endianness and act accordingly, or force the
> user-space app to always see little-endian registers by replacing
> memcpy_to/fromio calls with a loop over read;/writel,

You seem to assume that memcpy should do 32-bit reads/writes. Why not 
16-bit ones? That's why memcpy cannot do any byte swapping, because it 
can "theoretically" do 2 different types of it (16-bit and 32-bit), 
which is obviously not specified in memcpy interface.

-- 
Eugene

Re: Yosemite/440EP why are readl()/ioread32() setup to readlittle-endian?

[Peter Korsgaard]

>>>>> "Matt" == Matt Porter <mporter@kernel.crashing.org> writes:

 Matt> read*/write* and ioread*/iowrite* generate outbound little
 Matt> endian cycles on ALL arches, period.  They are intended
 Matt> only for PCI use and have generic names only because of
 Matt> the assumption that "all the world is a PC".

What is the preferred way of accessing non-PCI devices then? Direct
pointer access?

-- 
Bye, Peter Korsgaard

Re: Yosemite/440EP why are readl()/ioread32() setup to readlittle-endian?

[Kumar Gala]

On Wed, 1 Feb 2006, Peter Korsgaard wrote:

> >>>>> "Matt" == Matt Porter <mporter@kernel.crashing.org> writes:
> 
>  Matt> read*/write* and ioread*/iowrite* generate outbound little
>  Matt> endian cycles on ALL arches, period.  They are intended
>  Matt> only for PCI use and have generic names only because of
>  Matt> the assumption that "all the world is a PC".
> 
> What is the preferred way of accessing non-PCI devices then? Direct
> pointer access?

No direct pointer access is bad. On PPC You can use
in_be{8,16,32}/out_be{8,16,32}

- kumar

Re: Yosemite/440EP why are readl()/ioread32() setup to readlittle-endian?

[Matt Porter]

On Wed, Feb 01, 2006 at 06:54:09PM -0600, Kumar Gala wrote:
> On Wed, 1 Feb 2006, Peter Korsgaard wrote:
> 
> > >>>>> "Matt" == Matt Porter <mporter@kernel.crashing.org> writes:
> > 
> >  Matt> read*/write* and ioread*/iowrite* generate outbound little
> >  Matt> endian cycles on ALL arches, period.  They are intended
> >  Matt> only for PCI use and have generic names only because of
> >  Matt> the assumption that "all the world is a PC".
> > 
> > What is the preferred way of accessing non-PCI devices then? Direct
> > pointer access?
> 
> No direct pointer access is bad. On PPC You can use
> in_be{8,16,32}/out_be{8,16,32}

Ack. Also, it should be pointed out that there are countless
examples of PPC drivers where this is done properly. 4xx, 83xx,
85xx, etc. on-chip peripherals all do this since they are
naturally BE registers.

-Matt

Re: Yosemite/440EP why are readl()/ioread32() setup to readlittle-endian?

[Peter Korsgaard]

On 2/2/06, Kumar Gala <galak@kernel.crashing.org> wrote:
> > What is the preferred way of accessing non-PCI devices then? Direct
> > pointer access?
>
> No direct pointer access is bad. On PPC You can use
> in_be{8,16,32}/out_be{8,16,32}

What about arch independent drivers? Are there any generic approach
for this or do you have to stick to ugly #ifdefs to decide between
in_be32/inl ?

--
Bye, Peter Korsgaard

Re: Yosemite/440EP why are readl()/ioread32() setup to readlittle-endian?

[Eugene Surovegin]

On Thu, Feb 02, 2006 at 09:09:17AM +0100, Peter Korsgaard wrote:
> On 2/2/06, Kumar Gala <galak@kernel.crashing.org> wrote:
> > > What is the preferred way of accessing non-PCI devices then? Direct
> > > pointer access?
> >
> > No direct pointer access is bad. On PPC You can use
> > in_be{8,16,32}/out_be{8,16,32}
> 
> What about arch independent drivers? Are there any generic approach
> for this or do you have to stick to ugly #ifdefs to decide between
> in_be32/inl ?

I'm curious, could you give an example of such arch independent 
driver?

-- 
Eugene

Re: Yosemite/440EP why are readl()/ioread32() setup to readlittle-endian?

[Dale Farnsworth]

In article <20060202090827.GA12810@gate.ebshome.net> you write:
> On Thu, Feb 02, 2006 at 09:09:17AM +0100, Peter Korsgaard wrote:
> > On 2/2/06, Kumar Gala <galak@kernel.crashing.org> wrote:
> > > > What is the preferred way of accessing non-PCI devices then? Direct
> > > > pointer access?
> > >
> > > No direct pointer access is bad. On PPC You can use
> > > in_be{8,16,32}/out_be{8,16,32}
> > 
> > What about arch independent drivers? Are there any generic approach
> > for this or do you have to stick to ugly #ifdefs to decide between
> > in_be32/inl ?
> 
> I'm curious, could you give an example of such arch independent 
> driver?

Such #ifdefs are found in drivers/net/smc91x.h

-Dale

Re: Yosemite/440EP why are readl()/ioread32() setup to readlittle-endian?

[Matt Porter]

On Thu, Feb 02, 2006 at 09:09:17AM +0100, Peter Korsgaard wrote:
> On 2/2/06, Kumar Gala <galak@kernel.crashing.org> wrote:
> > > What is the preferred way of accessing non-PCI devices then? Direct
> > > pointer access?
> >
> > No direct pointer access is bad. On PPC You can use
> > in_be{8,16,32}/out_be{8,16,32}
> 
> What about arch independent drivers? Are there any generic approach
> for this or do you have to stick to ugly #ifdefs to decide between
> in_be32/inl ?

ioread*be()/iowrite*be() can be used for this. Since the generic
big endian accessors became a standard thing a little while back,
all the PPC on-chip drivers should move to that. It's just a matter
of time.

-Matt