PCI reading without endian conversion

PCI reading without endian conversion

[Matt Sealey]

Hi guys,

What's the correct way to read from PCI address space (basically it's
guaranteed to be non-coherent memory bar) without flipping bits like
ioread32() does?

I need to be able to copy a bank of registers from PCI address space
into a temporary buffer so I can compare them in userspace through
UIO. Because of the flipping and the difference between the original
kernel driver (which used ioread32() and therefore "saw" big endian)
and the userspace app (which has a direct view of the PCI space, and
therefore "sees" little endian) I decided to give userspace an
absolutely consistent little-endian view seeing as this may get ported
to ARM in the coming months.

I want to put as little code in there as possible and not laboriously
manually flip from my ioread32() big endian values to little endian
again (waste of time and code) if I can help it. Being able to read
the raw value would help a lot, and if I need to do calculations on a
small portion of the data then I can do the flips manually then (using
le32_to_cpu and cpu_to_le32 which will be a noop on ARM), reducing the
amount of porting I need to do in both kernel and userspace alike.

So, is there something like a direct ioread32le() or so, which will
not change behaviour across architectures, is present on ARM and PPC,
and will handle both PCI address space, and "normal" "ioremapped"
memory?

-- 
Matt Sealey <matt@genesi-usa.com>
Genesi, Manager, Developer Relations

Re: PCI reading without endian conversion

[Ira Snyder]

On Fri, Feb 20, 2009 at 12:57:36PM -0600, Matt Sealey wrote:
> Hi guys,
> 
> What's the correct way to read from PCI address space (basically it's
> guaranteed to be non-coherent memory bar) without flipping bits like
> ioread32() does?
> 
> I need to be able to copy a bank of registers from PCI address space
> into a temporary buffer so I can compare them in userspace through
> UIO. Because of the flipping and the difference between the original
> kernel driver (which used ioread32() and therefore "saw" big endian)
> and the userspace app (which has a direct view of the PCI space, and
> therefore "sees" little endian) I decided to give userspace an
> absolutely consistent little-endian view seeing as this may get ported
> to ARM in the coming months.
> 
> I want to put as little code in there as possible and not laboriously
> manually flip from my ioread32() big endian values to little endian
> again (waste of time and code) if I can help it. Being able to read
> the raw value would help a lot, and if I need to do calculations on a
> small portion of the data then I can do the flips manually then (using
> le32_to_cpu and cpu_to_le32 which will be a noop on ARM), reducing the
> amount of porting I need to do in both kernel and userspace alike.
> 
> So, is there something like a direct ioread32le() or so, which will
> not change behaviour across architectures, is present on ARM and PPC,
> and will handle both PCI address space, and "normal" "ioremapped"
> memory?
> 

I'm pretty sure memcpy_fromio() and memcpy_toio() will get you what you
want. They don't change byte ordering.

Ira

Re: PCI reading without endian conversion

[Matt Sealey]

On Fri, Feb 20, 2009 at 1:11 PM, Ira Snyder <iws@ovro.caltech.edu> wrote:
> On Fri, Feb 20, 2009 at 12:57:36PM -0600, Matt Sealey wrote:
>
> I'm pretty sure memcpy_fromio() and memcpy_toio() will get you what you
> want. They don't change byte ordering.

Are they guaranteed to only do 32-bit, aligned accesses?

I made some cheats on my CPLD to ignore byte enables and so on,
because it makes the design cleaner and easier to read (for students)
plus, saves a ton of logic cells. It's totally within the PCI
standard, but it means if you do a byte read memcpy() you get.. very
weird results (i.e. not great).

-- 
Matt Sealey <matt@genesi-usa.com>
Genesi, Manager, Developer Relations

Re: PCI reading without endian conversion

[Ira Snyder]

On Fri, Feb 20, 2009 at 02:05:08PM -0600, Matt Sealey wrote:
> On Fri, Feb 20, 2009 at 1:11 PM, Ira Snyder <iws@ovro.caltech.edu> wrote:
> > On Fri, Feb 20, 2009 at 12:57:36PM -0600, Matt Sealey wrote:
> >
> > I'm pretty sure memcpy_fromio() and memcpy_toio() will get you what you
> > want. They don't change byte ordering.
> 
> Are they guaranteed to only do 32-bit, aligned accesses?
> 

I don't think so. I certainly wouldn't count on anything better than a
byte-by-byte memcpy.

> I made some cheats on my CPLD to ignore byte enables and so on,
> because it makes the design cleaner and easier to read (for students)
> plus, saves a ton of logic cells. It's totally within the PCI
> standard, but it means if you do a byte read memcpy() you get.. very
> weird results (i.e. not great).
> 

Right, I understand how that works :)

Some usage of cscope shows that __raw_readl() might be what you want,
as well as __raw_writel() for writing. I'm not sure it is universally
available, but maybe they are.

The comment on PowerPC says "Non ordered and non-swapping "raw"
accessors". Looks about right. ARM's implementation uses them to
implement ioread32() and friends by adding byteswapping.

Hope it helps,
Ira

> -- 
> Matt Sealey <matt@genesi-usa.com>
> Genesi, Manager, Developer Relations
> 

Re: PCI reading without endian conversion

[Matt Sealey]

On Fri, Feb 20, 2009 at 3:07 PM, Ira Snyder <iws@ovro.caltech.edu> wrote:
> On Fri, Feb 20, 2009 at 02:05:08PM -0600, Matt Sealey wrote:
>> On Fri, Feb 20, 2009 at 1:11 PM, Ira Snyder <iws@ovro.caltech.edu> wrote:
>> > On Fri, Feb 20, 2009 at 12:57:36PM -0600, Matt Sealey wrote:
>> >
>> > I'm pretty sure memcpy_fromio() and memcpy_toio() will get you what you
>> > want. They don't change byte ordering.
>>
>> Are they guaranteed to only do 32-bit, aligned accesses?
>>
>
> I don't think so. I certainly wouldn't count on anything better than a
> byte-by-byte memcpy.
>
>> I made some cheats on my CPLD to ignore byte enables and so on,
>> because it makes the design cleaner and easier to read (for students)
>> plus, saves a ton of logic cells. It's totally within the PCI
>> standard, but it means if you do a byte read memcpy() you get.. very
>> weird results (i.e. not great).
>>
>
> Right, I understand how that works :)
>
> Some usage of cscope shows that __raw_readl() might be what you want,
> as well as __raw_writel() for writing. I'm not sure it is universally
> available, but maybe they are.
>
> The comment on PowerPC says "Non ordered and non-swapping "raw"
> accessors". Looks about right. ARM's implementation uses them to
> implement ioread32() and friends by adding byteswapping.

Am I correct in saying that cpu_to_le32 and le32_to_cpu are the
functions/macros I need to use to do byte swapping to make everything
go little endian (and back again when I read them back in the kernel)?

Or is there some cleverer way already implemented in the kernel?

-- 
Matt Sealey <matt@genesi-usa.com>
Genesi, Manager, Developer Relations

Re: PCI reading without endian conversion

[Ira Snyder]

On Fri, Feb 20, 2009 at 03:56:39PM -0600, Matt Sealey wrote:
> On Fri, Feb 20, 2009 at 3:07 PM, Ira Snyder <iws@ovro.caltech.edu> wrote:
> > On Fri, Feb 20, 2009 at 02:05:08PM -0600, Matt Sealey wrote:
> >> On Fri, Feb 20, 2009 at 1:11 PM, Ira Snyder <iws@ovro.caltech.edu> wrote:
> >> > On Fri, Feb 20, 2009 at 12:57:36PM -0600, Matt Sealey wrote:
> >> >
> >> > I'm pretty sure memcpy_fromio() and memcpy_toio() will get you what you
> >> > want. They don't change byte ordering.
> >>
> >> Are they guaranteed to only do 32-bit, aligned accesses?
> >>
> >
> > I don't think so. I certainly wouldn't count on anything better than a
> > byte-by-byte memcpy.
> >
> >> I made some cheats on my CPLD to ignore byte enables and so on,
> >> because it makes the design cleaner and easier to read (for students)
> >> plus, saves a ton of logic cells. It's totally within the PCI
> >> standard, but it means if you do a byte read memcpy() you get.. very
> >> weird results (i.e. not great).
> >>
> >
> > Right, I understand how that works :)
> >
> > Some usage of cscope shows that __raw_readl() might be what you want,
> > as well as __raw_writel() for writing. I'm not sure it is universally
> > available, but maybe they are.
> >
> > The comment on PowerPC says "Non ordered and non-swapping "raw"
> > accessors". Looks about right. ARM's implementation uses them to
> > implement ioread32() and friends by adding byteswapping.
> 
> Am I correct in saying that cpu_to_le32 and le32_to_cpu are the
> functions/macros I need to use to do byte swapping to make everything
> go little endian (and back again when I read them back in the kernel)?
> 
> Or is there some cleverer way already implemented in the kernel?
> 

I would say that the __raw_readl() reads in cpu order. If you wanted to
convert that to le32, you'd use cpu_to_le32().

Ira

Re: PCI reading without endian conversion

[Benjamin Herrenschmidt]

> > Am I correct in saying that cpu_to_le32 and le32_to_cpu are the
> > functions/macros I need to use to do byte swapping to make everything
> > go little endian (and back again when I read them back in the kernel)?
> > 
> > Or is there some cleverer way already implemented in the kernel?
> > 
> 
> I would say that the __raw_readl() reads in cpu order. If you wanted to
> convert that to le32, you'd use cpu_to_le32().

Beware that __raw_* forms also don't have memory barriers.

In general, you know what byte order your device uses (which is often
little endian) and you use the appropriate ioread32{be}. Now, for
userspace, you simply need to mimmic those accessors.

Ben.

Re: PCI reading without endian conversion

[Geert Uytterhoeven]

On Fri, 20 Feb 2009, Ira Snyder wrote:
> On Fri, Feb 20, 2009 at 03:56:39PM -0600, Matt Sealey wrote:
> > Am I correct in saying that cpu_to_le32 and le32_to_cpu are the
> > functions/macros I need to use to do byte swapping to make everything
> > go little endian (and back again when I read them back in the kernel)?
> > 
> > Or is there some cleverer way already implemented in the kernel?
> 
> I would say that the __raw_readl() reads in cpu order. If you wanted to
> convert that to le32, you'd use cpu_to_le32().

Well...

__raw_readl() does read in cpu order. But since you're reading from the PCI
bus, which is little endian, you actually read a little endian value. So you
should use le32_to_cpu() to convert it to a native value.

(yes, at the lowest level, cpu_to_le32() and le32_to_cpu() do the same thing
 on PPC: byte swapping 32-bit data).

With kind regards,

Geert Uytterhoeven
Software Architect

Sony Techsoft Centre Europe
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium

Phone:    +32 (0)2 700 8453
Fax:      +32 (0)2 700 8622
E-mail:   Geert.Uytterhoeven@sonycom.com
Internet: http://www.sony-europe.com/

A division of Sony Europe (Belgium) N.V.
VAT BE 0413.825.160 · RPR Brussels
Fortis · BIC GEBABEBB · IBAN BE41293037680010

Re: PCI reading without endian conversion

[Benjamin Herrenschmidt]

On Fri, 2009-02-20 at 12:57 -0600, Matt Sealey wrote:
> Hi guys,
> 
> What's the correct way to read from PCI address space (basically it's
> guaranteed to be non-coherent memory bar) without flipping bits like
> ioread32() does?

ioread32be() ? :-) But from what you say below, it seems the wrong
approach.

> I need to be able to copy a bank of registers from PCI address space
> into a temporary buffer so I can compare them in userspace through
> UIO. Because of the flipping and the difference between the original
> kernel driver (which used ioread32() and therefore "saw" big endian)
> and the userspace app (which has a direct view of the PCI space, and
> therefore "sees" little endian) I decided to give userspace an
> absolutely consistent little-endian view seeing as this may get ported
> to ARM in the coming months.

Your sentence above doesn't seem to make much sense to me ... Why don't
you just have your userspace use lwbrx and "see" the same thing as the
kernel ? Which would also happen to be the same thing as an ARM in LE
mode would see...

> I want to put as little code in there as possible and not laboriously
> manually flip from my ioread32() big endian values to little endian
> again (waste of time and code) if I can help it. Being able to read
> the raw value would help a lot, and if I need to do calculations on a
> small portion of the data then I can do the flips manually then (using
> le32_to_cpu and cpu_to_le32 which will be a noop on ARM), reducing the
> amount of porting I need to do in both kernel and userspace alike.
>
> So, is there something like a direct ioread32le() or so, which will
> not change behaviour across architectures, is present on ARM and PPC,
> and will handle both PCI address space, and "normal" "ioremapped"
> memory?

Little of what you say above make sense, you mix unrelated concepts and
all other weirdness mangled with purely false assumptions but from what
I can tell, what you should do is something along the line of:

 - kernel uses normal ioread32 on all platforms
 - userspace use lwbrx on powerpc and normal loads on LE platforms via
   some kind of macro you define for that

That will give you a consistent view accross the board.

Cheers,
Ben.