PPC PCI bus registers

PPC PCI bus registers

[Eddie Dawydiuk]

Hello,

I have a question regarding reading PCI bus registers from a user space 
application running on a PPC SBC. Seeing as though the PCI bus is little endian 
and PPC is big endian is it typical that one must perform a byte swap on all 16 
and 32 bit register reads?

I've found this is true on a custom board I am working on(with an FPGA connected 
via the PCI bus) and as a result I've added a byte swap command in busybox to 
accommodate this feature...

-- 
Best Regards,
________________________________________________________________
  Eddie Dawydiuk, Technologic Systems | voice:  (480) 837-5200
  16525 East Laser Drive              | fax:    (480) 837-5300
  Fountain Hills, AZ 85268            | web: www.embeddedARM.com

Re: PPC PCI bus registers

[Grant Likely]

On Wed, Sep 2, 2009 at 5:44 PM, Eddie Dawydiuk<eddie@embeddedarm.com> wrote:
> Hello,
>
> I have a question regarding reading PCI bus registers from a user space
> application running on a PPC SBC. Seeing as though the PCI bus is little
> endian and PPC is big endian is it typical that one must perform a byte swap
> on all 16 and 32 bit register reads?

Yes, this is correct.

g.

Re: PPC PCI bus registers

[Benjamin Herrenschmidt]

On Wed, 2009-09-02 at 16:44 -0700, Eddie Dawydiuk wrote:
> Hello,
> 
> I have a question regarding reading PCI bus registers from a user space 
> application running on a PPC SBC. Seeing as though the PCI bus is little endian 
> and PPC is big endian is it typical that one must perform a byte swap on all 16 
> and 32 bit register reads?
> 
> I've found this is true on a custom board I am working on(with an FPGA connected 
> via the PCI bus) and as a result I've added a byte swap command in busybox to 
> accommodate this feature...

Note that powerpc has efficient load/store reverse instructions that
perform the byteswap for you. We use them for IOs in the kernel for
example.

Also, if you're going to access a PCI device directly, beware of other
issues such as ordering. PPC is an out of order architecture, you need
to ensure you add the appropriate memory barriers if you want to ensure
you accesses are done in the order you write them in your program.

For "standard" stuff that doesn't involve DMA or locks, an eieio after
both MMIO loads and stores should do the trick.

If you need to order vs. DMA and/or locks, you may want to look at what
the kernel does in io.h

Cheers,
Ben.

Re: PPC PCI bus registers

[Eddie Dawydiuk]

Benjamin,

> Also, if you're going to access a PCI device directly, beware of other
> issues such as ordering. PPC is an out of order architecture, you need
> to ensure you add the appropriate memory barriers if you want to ensure
> you accesses are done in the order you write them in your program.
> 
> For "standard" stuff that doesn't involve DMA or locks, an eieio after
> both MMIO loads and stores should do the trick.

I'm not sure I understand. To clarify I have an FPGA connected via the PCI bus 
which implements several peripherals, I've implemented device drivers for. 
Currently I am calling ioremap() to get a virtual address corresponding to the 
PCI devices. Then I use ___raw_writeN / ___raw_readN for reading/writing data 
via the PCI bus to the FPGA registers. From looking at io.h I believe this 
method is safe with regard to out of order execution.

"* ioremap is the standard one and provides non-cacheable guarded mappings
*   and can be hooked by the platform via ppc_md "

Can you verify if my understanding is correct, or let me know if I need to add 
memory barriers?

-- 
Best Regards,
________________________________________________________________
  Eddie Dawydiuk, Technologic Systems | voice:  (480) 837-5200
  16525 East Laser Drive              | fax:    (480) 837-5300
  Fountain Hills, AZ 85268            | web: www.embeddedARM.com

Re: PPC PCI bus registers

[Benjamin Herrenschmidt]

On Wed, 2009-09-16 at 11:58 -0700, Eddie Dawydiuk wrote:
> I'm not sure I understand. To clarify I have an FPGA connected via the
> PCI bus 
> which implements several peripherals, I've implemented device drivers
> for. 
> Currently I am calling ioremap() to get a virtual address
> corresponding to the 
> PCI devices. Then I use ___raw_writeN / ___raw_readN for
> reading/writing data 
> via the PCI bus to the FPGA registers. From looking at io.h I believe
> this 
> method is safe with regard to out of order execution.
> 
> "* ioremap is the standard one and provides non-cacheable guarded
> mappings
> *   and can be hooked by the platform via ppc_md "
> 
> Can you verify if my understanding is correct, or let me know if I
> need to add 
> memory barriers?
> 

Out of order execution != out of order storage. ioremap() will give you
guarded space which means it cannot be speculatively accessed for
example, and you do get -some- guarantees but not that your stores are
going to hit the device in order, nor that your loads are going to be
performed until the CPU actually use the result of the load, which can
be delayed beyond a store.

Cheers,
Ben.

Re: PPC PCI bus registers

[Eddie Dawydiuk]

Benjamin,

> Out of order execution != out of order storage. ioremap() will give you
> guarded space which means it cannot be speculatively accessed for
> example, and you do get -some- guarantees but not that your stores are
> going to hit the device in order, nor that your loads are going to be
> performed until the CPU actually use the result of the load, which can
> be delayed beyond a store.

Interesting. So IIUC if I were to use ioremap and then bit bang bus cycles using 
readN() / writeN() I would need to add memory barrier to ensure the order of the 
load and store operations does not change. Is my understanding correct? If so 
what are the recommended macros/functions for adding memory barriers, or are 
there preferred functions to use in place of ioremap() and readN()/writeN?

PS I've looked in Documentation/ in the 2.6.30 Linux kernel tree for more 
information, but I was unable to find any. If there are any example drivers or 
documentation that you know of on this topic I'd be quite interested if you 
could point me to it.

Thanks for your patience and time you've been quite helpful.

-- 
Best Regards,
________________________________________________________________
  Eddie Dawydiuk, Technologic Systems | voice:  (480) 837-5200
  16525 East Laser Drive              | fax:    (480) 837-5300
  Fountain Hills, AZ 85268            | web: www.embeddedARM.com

Re: PPC PCI bus registers

[Benjamin Herrenschmidt]

On Wed, 2009-09-16 at 15:11 -0700, Eddie Dawydiuk wrote:
> Benjamin,
> 
> > Out of order execution != out of order storage. ioremap() will give you
> > guarded space which means it cannot be speculatively accessed for
> > example, and you do get -some- guarantees but not that your stores are
> > going to hit the device in order, nor that your loads are going to be
> > performed until the CPU actually use the result of the load, which can
> > be delayed beyond a store.
> 
> Interesting. So IIUC if I were to use ioremap and then bit bang bus cycles using 
> readN() / writeN() I would need to add memory barrier to ensure the order of the 
> load and store operations does not change. Is my understanding correct? 

Only if you use the __raw variants. readN/writeN already provide full
barriers, including barriers against leakage outside of locks and
barriers vs. cacheable accesses (MMIO vs. DMA) and thus are slow.

If you use the raw variants, it then depends whether your code is going
to be ppc specific, in which case you can manually use things like
eieio() etc... or not in which case ... it's harder :-) You can always
use rmb/wmb/mb but those are more heavyweight than just an eieio

There's iobarrier_r/w/rw, dunno if those are generic tho.

> If so 
> what are the recommended macros/functions for adding memory barriers, or are 
> there preferred functions to use in place of ioremap() and readN()/writeN?

ioremap + readN / writeN will give you fully ordered behaviour. It's
only when you try to be sneaky or speedy and use the __raw  variants
that you have to be more careful.

> PS I've looked in Documentation/ in the 2.6.30 Linux kernel tree for more 
> information, but I was unable to find any. If there are any example drivers or 
> documentation that you know of on this topic I'd be quite interested if you 
> could point me to it.
> 
> Thanks for your patience and time you've been quite helpful.

Cheers,
Ben.

Re: PPC PCI bus registers

[Eddie Dawydiuk]

Benjamin,

> ioremap + readN / writeN will give you fully ordered behaviour. It's
> only when you try to be sneaky or speedy and use the __raw  variants
> that you have to be more careful.

You've been most helpful, thanks.

-- 
Best Regards,
________________________________________________________________
  Eddie Dawydiuk, Technologic Systems | voice:  (480) 837-5200
  16525 East Laser Drive              | fax:    (480) 837-5300
  Fountain Hills, AZ 85268            | web: www.embeddedARM.com