Memory mapping PCI memory region to user space

Memory mapping PCI memory region to user space

[Wyse, Chris]

[-- Attachment #1: Type: text/plain, Size: 1871 bytes --]

Hi,
 
I'm trying to map a PCI memory region 1 into user space from my driver
(PPC440GX, Linux 2.6.10).  Here's the mmap routine of the driver that
I'm using:
 
 
staticfun
int phob_mmap(struct file *filp, struct vm_area_struct *vma)
{
    struct phob_file *fptr = filp->private_data;
    struct phob_struct *dev = fptr->dev;
    struct phob_region *rptr = &dev->region[1];
    phys_addr_t physAddr;
 

    physAddr = rptr->address + PPC44x_PCIMEM_PAGE;
    if (io_remap_page_range(vma, vma->vm_start,
                            physAddr,
                            rptr->len,
                            vma->vm_page_prot))
        return -EAGAIN;
}


Am I using the correct routine to do this?
 
When I use this routine, the pfn_pte (pgtable.h): 
 
#define pfn_pte(pfn, prot)      __pte(((pfn) << PAGE_SHIFT) |
pgprot_val(prot))
 
macro shifted out the high order bits of the pfn, which I didn't think
was correct, so I changed it to:

#define pfn_pte(pfn, prot)      __pte((((unsigned long long) (pfn)) <<
PAGE_SHIFT) | pgprot_val(prot))
 
For reference, the call stack up to the pfn_pte macro is:
 
remap_pte_range()
remap_pmd_range()
remap_pfn_range()
io_remap_page_range()
 
After changing the macro, the value seems to be saved properly, and the
TLB gets updated properly when I get a TLB miss.  However, when if I try
to write to the PCI from user space, I get an error from do_wp_page()
stating that it is a bogus pfn.  If I try to read from user space, the
processor goes out to lunch and I need to reboot.
 
Am I missing something obvious?  Anyone have any ideas on how to do this
and/or ways to debug it?
 
Thanks.


Chris Wyse
Member of Technical Staff
Embedded Technologies
860-749-1556 office
860-978-0849 cell
413-778-9101 fax
http://www.windriver.com <http://www.windriver.com/> 
  

 

[-- Attachment #2: Type: text/html, Size: 6392 bytes --]

Re: Memory mapping PCI memory region to user space

[Kumar Gala]

On Mar 23, 2006, at 8:21 AM, Wyse, Chris wrote:

> Hi,
>
> I'm trying to map a PCI memory region 1 into user space from my  
> driver (PPC440GX, Linux 2.6.10).  Here's the mmap routine of the  
> driver that I'm using:

Why don't use the mmap file exposed by sysfs so you dont have to  
write your own code?

See Documentation/filesystems/sysfs-pci.txt.  But effectively down  
under /sys/bus/pci/devices/[domain:bus:dev:func]/ you will get  
resource[0..N-1] that corresponds to each BAR on the device.  This is  
a mmap file to access that region.

- kumar

>
>
> staticfun
> int phob_mmap(struct file *filp, struct vm_area_struct *vma)
> {
>     struct phob_file *fptr = filp->private_data;
>     struct phob_struct *dev = fptr->dev;
>     struct phob_region *rptr = &dev->region[1];
>     phys_addr_t physAddr;
>
>     physAddr = rptr->address + PPC44x_PCIMEM_PAGE;
>     if (io_remap_page_range(vma, vma->vm_start,
>                             physAddr,
>                             rptr->len,
>                             vma->vm_page_prot))
>         return -EAGAIN;
> }
> Am I using the correct routine to do this?
>
> When I use this routine, the pfn_pte (pgtable.h):
>
> #define pfn_pte(pfn, prot)      __pte(((pfn) << PAGE_SHIFT) |  
> pgprot_val(prot))
>
> macro shifted out the high order bits of the pfn, which I didn't  
> think was correct, so I changed it to:
>
> #define pfn_pte(pfn, prot)      __pte((((unsigned long long) (pfn))  
> << PAGE_SHIFT) | pgprot_val(prot))
>
> For reference, the call stack up to the pfn_pte macro is:
>
> remap_pte_range()
> remap_pmd_range()
> remap_pfn_range()
> io_remap_page_range()
>
> After changing the macro, the value seems to be saved properly, and  
> the TLB gets updated properly when I get a TLB miss.  However, when  
> if I try to write to the PCI from user space, I get an error from  
> do_wp_page() stating that it is a bogus pfn.  If I try to read from  
> user space, the processor goes out to lunch and I need to reboot.
>
> Am I missing something obvious?  Anyone have any ideas on how to do  
> this and/or ways to debug it?
>
> Thanks.
>
> Chris Wyse
> Member of Technical Staff
> Embedded Technologies
> 860-749-1556 office
> 860-978-0849 cell
> 413-778-9101 fax
> http://www.windriver.com
>
>
>
> _______________________________________________
> Linuxppc-embedded mailing list
> Linuxppc-embedded@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-embedded

Re: Memory mapping PCI memory region to user space

[David Hawkins]

> Why don't use the mmap file exposed by sysfs so you dont have to  
> write your own code?
> 
> See Documentation/filesystems/sysfs-pci.txt.  But effectively down  
> under /sys/bus/pci/devices/[domain:bus:dev:func]/ you will get  
> resource[0..N-1] that corresponds to each BAR on the device.  This is  
> a mmap file to access that region.

Hi Kumar,

Most likely he didn't know about it, I didn't :)

When I was testing the Yosemite board as the host, I found
that I could set the endian flag on the mmapped page, which
then made the PCI device registers read as 32-bit quantities
read back with the same layout under both x86 and PPC
hosts.

So, if someone planned on using that 'feature', that would
be a reason for writing your own mmap() implementation.
I didn't see any option in the sysfs file for setting that
page flag (but, I didn't look around in other sysfs files).

Thanks for the pointer to the sysfs node!

Cheers
Dave

Re: Memory mapping PCI memory region to user space

[Kumar Gala]

On Mar 23, 2006, at 11:12 AM, David Hawkins wrote:

>
>
>> Why don't use the mmap file exposed by sysfs so you dont have to   
>> write your own code?
>> See Documentation/filesystems/sysfs-pci.txt.  But effectively  
>> down  under /sys/bus/pci/devices/[domain:bus:dev:func]/ you will  
>> get  resource[0..N-1] that corresponds to each BAR on the device.   
>> This is  a mmap file to access that region.
>
> Hi Kumar,
>
> Most likely he didn't know about it, I didn't :)
>
> When I was testing the Yosemite board as the host, I found
> that I could set the endian flag on the mmapped page, which
> then made the PCI device registers read as 32-bit quantities
> read back with the same layout under both x86 and PPC
> hosts.

Hmm, I guess I would handle this like how the reset of the kernel  
handle is with the io routines handling the swapping.  Not sure if  
there is any advantage to using the endian flag.  I guess if you have  
something you are treating as just memory there would be.

> So, if someone planned on using that 'feature', that would
> be a reason for writing your own mmap() implementation.
> I didn't see any option in the sysfs file for setting that
> page flag (but, I didn't look around in other sysfs files).

There isn't a sysfs flag for the endianness page attribute since  
thats a PPC book-e specific feature.  We could possible expand things  
to support it but, I've been trying to actively avoid using the 'E' bit.

> Thanks for the pointer to the sysfs node!

No problem, its what we're here for :)

- kumar

Re: Memory mapping PCI memory region to user space

[Mark Chambers]

>>
>> When I was testing the Yosemite board as the host, I found
>> that I could set the endian flag on the mmapped page, which
>> then made the PCI device registers read as 32-bit quantities
>> read back with the same layout under both x86 and PPC
>> hosts.
>
> Hmm, I guess I would handle this like how the reset of the kernel
> handle is with the io routines handling the swapping.  Not sure if
> there is any advantage to using the endian flag.  I guess if you have
> something you are treating as just memory there would be.
>
>> So, if someone planned on using that 'feature', that would
>> be a reason for writing your own mmap() implementation.
>> I didn't see any option in the sysfs file for setting that
>> page flag (but, I didn't look around in other sysfs files).
>
> There isn't a sysfs flag for the endianness page attribute since
> thats a PPC book-e specific feature.  We could possible expand things
> to support it but, I've been trying to actively avoid using the 'E' bit.
>

Framebuffer applications could care about this.  Also, on the MPC8247
(maybe others, but I know about this one) you can make the PCI space
big- or little-endian.

Seems to me this is really not an endian issue but a PCI implementation
issue.  PCI is actually endian-agnostic, so we're really 'fixing' the 
Freescale
PCI implementation with endian swapping.

Mark Chambers 

Re: Memory mapping PCI memory region to user space

[David Hawkins]

> 
> PCI is actually endian-agnostic, so we're really 'fixing' the 
> Freescale PCI implementation with endian swapping.
> 

Hi Mark,

Its not agnostic, its little-endian.

The configuration space layout of parameters larger than a
byte are defined in little endian format.

The wiring on the PCI bus is little-endian, and 32-bit
quantities on the bus are represented by a logic analyzer
in little endian format.

But once you start talking about BAR regions, then yeah,
they're your bytes, do with them what you will. My hardware
uses a little-endian TI DSP, and an x86 host, so I
had been using compatible machines.

But the PLX PCI-9054 PCI chipset exposes BAR[0] and BAR[1]
as control registers that are little-endian format.

Conversely, we also use some Janz CAN cards, and their
messaging protocol over PCI uses big-endian format
structures in their protocol.

Dave

Re: Memory mapping PCI memory region to user space

[Mark Chambers]

> 
>> 
>> PCI is actually endian-agnostic, so we're really 'fixing' the 
>> Freescale PCI implementation with endian swapping.
>> 
> 
> Hi Mark,
> 
> Its not agnostic, its little-endian.
> 

Ok, I should be a little more specific.  Yes, I/O space is
little endian, and any configuration registers and such are
little endian.  But memory space is strictly 32 bit as far
as PCI is concerned. (forgetting 64 bit PCI for the moment)
 The two lower bits of address are not used, and there 
is no required correlation of byte enables to those missing 
address bits.

So, the point is, Freescale swaps bytes between its internal
bus and PCI.  Other processors (like TI DSPs) do not.  I
don't know that one method is necessarily right, but the fact
that we have this discussion periodically suggests that 
Freescale's method is not the best.

This might be an academic point, but I think it does help to
see the distinction.  To talk to a device over PCI you must
know how both ends map their internal buss(es) to PCI,
and it's not directly a big/little endian issue.

Mark

Re: Memory mapping PCI memory region to user space

[David Hawkins]

Hi Mark,

> Ok, I should be a little more specific.

Ok :)

> Yes, I/O space is little endian, and any configuration
> registers and such are little endian.  But memory space
> is strictly 32 bit as far as PCI is concerned.
> (forgetting 64 bit PCI for the moment) The two lower
> bits of address are not used, and there is no required 
> correlation of byte enables to those missing address bits.
> 
> So, the point is, Freescale swaps bytes between its internal
> bus and PCI.  Other processors (like TI DSPs) do not.  I
> don't know that one method is necessarily right, but the fact
> that we have this discussion periodically suggests that Freescale's 
> method is not the best.

Hmm, I'd have to look on the PCI bus with the analyzer to
confirm this ... but I do recall seeing a mapping of
the 128-bit PLB to PCI bus in the 440EP manual, so
you're probably right.

The PLX PCI-9054 has an 'endianness' option like this too.
I believe its so you can use a PPC on the local bus, and
swap bytes when the are written onto the PCI bus.
Sounds like the Freescale SoC bridges have just hardcoded
that type of implementation.

> This might be an academic point, but I think it does help to
> see the distinction.  To talk to a device over PCI you must
> know how both ends map their internal buss(es) to PCI,
> and it's not directly a big/little endian issue.

Its nice to be aware of these subtle differences.

Thanks for the discussion.

Dave

Re: Memory mapping PCI memory region to user space

[David Hawkins]

Hi Kumar,

>> When I was testing the Yosemite board as the host, I found
>> that I could set the endian flag on the mmapped page, which
>> then made the PCI device registers read as 32-bit quantities
>> read back with the same layout under both x86 and PPC
>> hosts.
> 
> Hmm, I guess I would handle this like how the reset of the kernel  
> handle is with the io routines handling the swapping.  Not sure if  
> there is any advantage to using the endian flag.  I guess if you have  
> something you are treating as just memory there would be.

I haven't used the feature, I just tested it to see what it did.

The application case I thought of was this; the PCI boards I built
(that I am revising, and replacing the DSP with a PPC) have an
8MB PCI region that I can mmap from the host. I have a test
suite that runs from the host that manipulates registers on the boards
to download FPGAs etc. When the boards are used in a real system,
the onboard DSP is generally used, and the host just talks to
the DSP.

However, for the test suite, if I have a header with definitions
like:

#define CONTROL_FPGA_ENABLE   (1 << 0)
#define CONTROL_FPGA_DONE_BIT (1 << 1)

that correspond to bits in a 32-bit PCI mmapped register. Then
code in the user-space test suite that did something like

   pci_addr[CONTROL_OFFSET] |= CONTROL_FPGA_ENABLE;

would instead need to be re-written, eg.,

   write_le32(&pci_addr[CONTROL_OFFSET], CONTROL_FPGA_ENABLE);

to be portable.

I definitely agree that this is how kernel-level code should be
written, but user-space code ... well, if I want to reuse code
already written, setting the page endian flag and reusing the
code would seem like the way to go. (This isn't what I need
to do, since my host will still be an x86, the PPC will
be a target device, but I still need to think about the
endian issues).

Now of course that I have seen the consequences of my coding,
I'll be more careful to deal with endianness more appropriately.

Its a tricky trade-off though. I could define control ioctl's
that hide all the endianness issues ... but then the driver just
gets bigger. I think the appropriate solution for the user-space
test code would be to use CPU-to-little-endian routines, and
wrap the lot in a re-usable library that the test suite
links against.

> There isn't a sysfs flag for the endianness page attribute since  thats 
> a PPC book-e specific feature.  We could possible expand things  to 
> support it but, I've been trying to actively avoid using the 'E' bit.

Ok, I haven't received the 8349E board that I am waiting
on, so I hadn't spotted that the PAGE_ENDIAN flag was
Book E specific.

Thanks for your insight.

Cheers
Dave

Re: Memory mapping PCI memory region to user space

[Phil Nitschke]

On Thu, 2006-03-23 at 09:44 -0600, Kumar Gala wrote:
> On Mar 23, 2006, at 8:21 AM, Wyse, Chris wrote:
> 
> > Hi,
> >
> > I'm trying to map a PCI memory region 1 into user space from my  
> > driver (PPC440GX, Linux 2.6.10).  Here's the mmap routine of the  
> > driver that I'm using:
> 
> Why don't use the mmap file exposed by sysfs so you dont have to  
> write your own code?
> 
> See Documentation/filesystems/sysfs-pci.txt.  But effectively down  
> under /sys/bus/pci/devices/[domain:bus:dev:func]/ you will get  
> resource[0..N-1] that corresponds to each BAR on the device.  This is  
> a mmap file to access that region.

I have some custom hardware that appears on the PCI bus as follows:

bash-3.00# lspci -vv
00:01.0 Class 0680: 1172:0004 (rev 01)
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B-
	Status: Cap- 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort+
<TAbort- <MAbort- >SERR- <PERR-
	Latency: 128, Cache Line Size 08
	Interrupt: pin A routed to IRQ 71
	Region 0: Memory at 000000009ffff000 (32-bit, non-prefetchable)
[size=4K]
	Region 1: Memory at 000000009fc00000 (32-bit, non-prefetchable)
[size=2M]

But when I try to access resource0 or resource1, I get a read error.
What characteristic of the device or driver determines whether it will
allow mmap-ing?

(I've written the driver for this device myself.)

-- 
Phil

Re: Memory mapping PCI memory region to user space

[David Hawkins]

> I have some custom hardware that appears on the PCI bus as follows:
> 
> bash-3.00# lspci -vv
> 00:01.0 Class 0680: 1172:0004 (rev 01)
> 	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B-
> 	Status: Cap- 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort+
> <TAbort- <MAbort- >SERR- <PERR-
> 	Latency: 128, Cache Line Size 08
> 	Interrupt: pin A routed to IRQ 71
> 	Region 0: Memory at 000000009ffff000 (32-bit, non-prefetchable)
> [size=4K]
> 	Region 1: Memory at 000000009fc00000 (32-bit, non-prefetchable)
> [size=2M]
> 
> But when I try to access resource0 or resource1, I get a read error.
> What characteristic of the device or driver determines whether it will
> allow mmap-ing?
> 
> (I've written the driver for this device myself.)
> 

Hi Phil,

The board tells you you have 4K of non-prefetchable memory in two BAR
regions. Which is what you've already determined I am sure.

Try building an insmodding the pci_io driver I posted, it'll pick
up the two BAR regions and allow you to access them, and you can
add more debugging comments to that code fairly easily.

If things don't work then, you probably have a hardware error and
would benefit from looking at the PCI bus with a logic analyzer to
see if the device is generating an error.

http://www.ovro.caltech.edu/~dwh/pci_io.tar.gz

Regards,
Dave

Re: Memory mapping PCI memory region to user space

[Phil Nitschke]

On Mon, 2006-03-27 at 08:05 -0800, David Hawkins wrote:
>
> Try building an insmodding the pci_io driver I posted, it'll pick
> up the two BAR regions and allow you to access them, and you can
> add more debugging comments to that code fairly easily.

> http://www.ovro.caltech.edu/~dwh/pci_io.tar.gz

Hi Dave,

I tried your program briefly, but (I guess) since I'm not using udev,
the program failed rather early, and I didn't persist for too long.
I've attached a log of the session below.  

(I used it on a commercial VME carrier board with a PmPPC7448 PMC module
and a PMC FPDP module, and a recent 2.6.16 kernel.org kernel built for
PPC. I used the ELDK4 NFS filesystem.)

-- 
Phil Nitschke <Phil.Nitschke@avalon.com.au>
Avalon Systems Pty Ltd


Linux 2.6.16-pmppc744x (arty5) (13:01 on Tuesday, 28 March 2006)

login: root
Last login: Tue Mar 28 13:00:32 from 192.168.1.14
bash-3.00# mount
/dev/nfs on / type nfs (rw)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
bash-3.00# lsmod
Module                  Size  Used by
bash-3.00# lspci -v 
00:01.0 Class 0680: 10e3:0148 (rev 01)
	Subsystem: 10e3:0000
	Flags: bus master, 66Mhz, medium devsel, latency 128, IRQ 71
	Memory at 000000009ffff000 (64-bit, non-prefetchable) [size=4K]
	Capabilities: [40] PCI-X non-bridge device.

00:02.0 Class 1100: 129a:dd11 (rev 01)
	Subsystem: 129a:dd11
	Flags: bus master, medium devsel, latency 128, IRQ 72
	Memory at 000000009f000000 (32-bit, prefetchable) [size=8M]
	Memory at 000000009e800000 (32-bit, non-prefetchable) [size=8M]

bash-3.00# mknod /dev/pci_00:02.0_0 c 254 0
bash-3.00# insmod /lib/modules/2.6.16-pmppc744x/misc/pci_io.ko major=254
bash-3.00# tail -6 /var/log/messages 
Mar 28 13:04:24 arty5 ntpd[236]: kernel time sync disabled 0041
Mar 28 13:04:51 arty5 kernel: pci_io: <init>
Mar 28 13:04:51 arty5 kernel: pci_io: register the PCI driver
Mar 28 13:04:51 arty5 kernel: pci_io: PCI registration succeeded
(returned 0)
Mar 28 13:04:51 arty5 kernel: pci_io: module loaded
Mar 28 13:05:29 arty5 ntpd[236]: kernel time sync enabled 0001
bash-3.00# ls -la  /sys/module/pci_io/parameters/
total 0
drwxr-xr-x  2 root root    0 Mar 28 13:04 .
drwxr-xr-x  4 root root    0 Mar 28 13:04 ..
-r--r--r--  1 root root 4096 Mar 28 13:22 debug
-r--r--r--  1 root root 4096 Mar 28 13:22 major
-r--r--r--  1 root root 4096 Mar 28 13:22 search
bash-3.00# cat /sys/module/pci_io/parameters/debug
1
bash-3.00# cat /sys/module/pci_io/parameters/major 
254
bash-3.00# cat /sys/module/pci_io/parameters/search 
0
bash-3.00# pci_debug -s 00:02.0 -e
Open device /dev/pci_00:02.0_0
Device open failed: errno 6, No such device or address
bash-3.00# rmmod pci_io
bash-3.00# insmod /lib/modules/2.6.16-pmppc744x/misc/pci_io.ko major=254
search=1
bash-3.00# tail -6 /var/log/messages 
Mar 28 13:24:03 arty5 kernel: pci_io: <init>
Mar 28 13:24:03 arty5 kernel: pci_io: search for 1 specified PCI IDs
Mar 28 13:24:03 arty5 kernel: pci_io: device 10b5:10b5 was not found
Mar 28 13:24:03 arty5 kernel: pci_io: register the PCI driver
Mar 28 13:24:03 arty5 kernel: pci_io: PCI registration succeeded
(returned 0)
Mar 28 13:24:03 arty5 kernel: pci_io: module loaded
bash-3.00# pci_debug -s 00:02.0 -e
Open device /dev/pci_00:02.0_0
Device open failed: errno 6, No such device or address

Re: Memory mapping PCI memory region to user space

[David Hawkins]

Phil Nitschke wrote:
> On Mon, 2006-03-27 at 08:05 -0800, David Hawkins wrote:
> 
>>Try building an insmodding the pci_io driver I posted, it'll pick
>>up the two BAR regions and allow you to access them, and you can
>>add more debugging comments to that code fairly easily.
> 
> 
>>http://www.ovro.caltech.edu/~dwh/pci_io.tar.gz
> 
> 
> Hi Dave,
> 
> I tried your program briefly, but (I guess) since I'm not using udev,
> the program failed rather early, and I didn't persist for too long.
> I've attached a log of the session below.  
> 
> (I used it on a commercial VME carrier board with a PmPPC7448 PMC module
> and a PMC FPDP module, and a recent 2.6.16 kernel.org kernel built for
> PPC. I used the ELDK4 NFS filesystem.)
> 

Sorry, I should have metioned. If udev is not installed then
the scripts that create the /dev nodes do not exist, so
you need to create them manually.

I don't see any probe calls in the driver log, so I'm not sure
that the driver has been told to register for that device ID.
You can either add your device ID to the pci IDs list in the
driver, or echo to a sysfs node ... let me see ... yeah

echo "129a:dd11" > /sys/bus/pci/drivers/pci_io/new_id

should do the trick.

Then you'll see probe callbacks and comments about
what major and start minor is assigned to each device.

Then create your dev nodes, eg major=254, minor=0, count=3

mknod /dev/pci_00:02.0_0 c 254 0
mknod /dev/pci_00:02.0_1 c 254 1
mknod /dev/pci_00:02.0_2 c 254 2

which is what you were trying already.

Give that a shot.

Dave

Re: Memory mapping PCI memory region to user space

[Phil Nitschke]

On Mon, 2006-03-27 at 20:55 -0800, David Hawkins wrote:

> I don't see any probe calls in the driver log, so I'm not sure
> that the driver has been told to register for that device ID.
> You can either add your device ID to the pci IDs list in the
> driver, 

OK, I added the device ID(s).

> or echo to a sysfs node ... let me see ... yeah
> 
> echo "129a:dd11" > /sys/bus/pci/drivers/pci_io/new_id
> 
> should do the trick.

That didn't work for me.  (Permission denied, IIRC)

> Then you'll see probe callbacks and comments about
> what major and start minor is assigned to each device.
> 
> Then create your dev nodes, eg major=254, minor=0, count=3
> 
> mknod /dev/pci_00:02.0_0 c 254 0
> mknod /dev/pci_00:02.0_1 c 254 1
> mknod /dev/pci_00:02.0_2 c 254 2
> 
> which is what you were trying already.

Much better, thanks.  I might be able to use your program/driver as a
comparison to my own driver, at least as far as the mmap() goes, because
mine is broken and yours isn't!

-- 
Phil Nitschke <Phil.Nitschke@avalon.com.au>
Avalon Systems Pty Ltd

Re: Memory mapping PCI memory region to user space

[David Hawkins]

Hey Phil,
> 
> Much better, thanks.  I might be able to use your program/driver as a
> comparison to my own driver, at least as far as the mmap() goes, because
> mine is broken and yours isn't!
> 

Great! Good to hear.

Dave

Re: Memory mapping PCI memory region to user space

[Kumar Gala]

On Mar 27, 2006, at 2:02 AM, Phil Nitschke wrote:

> On Thu, 2006-03-23 at 09:44 -0600, Kumar Gala wrote:
>> On Mar 23, 2006, at 8:21 AM, Wyse, Chris wrote:
>>
>>> Hi,
>>>
>>> I'm trying to map a PCI memory region 1 into user space from my
>>> driver (PPC440GX, Linux 2.6.10).  Here's the mmap routine of the
>>> driver that I'm using:
>>
>> Why don't use the mmap file exposed by sysfs so you dont have to
>> write your own code?
>>
>> See Documentation/filesystems/sysfs-pci.txt.  But effectively down
>> under /sys/bus/pci/devices/[domain:bus:dev:func]/ you will get
>> resource[0..N-1] that corresponds to each BAR on the device.  This is
>> a mmap file to access that region.
>
> I have some custom hardware that appears on the PCI bus as follows:
>
> bash-3.00# lspci -vv
> 00:01.0 Class 0680: 1172:0004 (rev 01)
> 	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B-
> 	Status: Cap- 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort+
> <TAbort- <MAbort- >SERR- <PERR-
> 	Latency: 128, Cache Line Size 08
> 	Interrupt: pin A routed to IRQ 71
> 	Region 0: Memory at 000000009ffff000 (32-bit, non-prefetchable)
> [size=4K]
> 	Region 1: Memory at 000000009fc00000 (32-bit, non-prefetchable)
> [size=2M]
>
> But when I try to access resource0 or resource1, I get a read error.
> What characteristic of the device or driver determines whether it will
> allow mmap-ing?
>
> (I've written the driver for this device myself.)

Nothing special beyond normal unix perms on the resource[0..n] files  
to my knowledge.  When you say you get a read error what exactly does  
that mean?

- kumar

Re: Memory mapping PCI memory region to user space

[Phil Nitschke]

[-- Attachment #1: Type: text/plain, Size: 1960 bytes --]

On Mon, 2006-03-27 at 10:18 -0600, Kumar Gala wrote:
> On Mar 27, 2006, at 2:02 AM, Phil Nitschke wrote:
> 
> > On Thu, 2006-03-23 at 09:44 -0600, Kumar Gala wrote:
> >> On Mar 23, 2006, at 8:21 AM, Wyse, Chris wrote:
> >>
> >>> Hi,
> >>>
> >>> I'm trying to map a PCI memory region 1 into user space from my
> >>> driver (PPC440GX, Linux 2.6.10).  Here's the mmap routine of the
> >>> driver that I'm using:
> >>
> >> Why don't use the mmap file exposed by sysfs so you dont have to
> >> write your own code?
> >>
> >> See Documentation/filesystems/sysfs-pci.txt.  But effectively down
> >> under /sys/bus/pci/devices/[domain:bus:dev:func]/ you will get
> >> resource[0..N-1] that corresponds to each BAR on the device.  This is
> >> a mmap file to access that region.
> >
> > I have some custom hardware that appears on the PCI bus as follows:
> >
> > bash-3.00# lspci -vv
> > 00:01.0 Class 0680: 1172:0004 (rev 01)
> > 	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> > Stepping- SERR- FastB2B-
> > 	Status: Cap- 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort+
> > <TAbort- <MAbort- >SERR- <PERR-
> > 	Latency: 128, Cache Line Size 08
> > 	Interrupt: pin A routed to IRQ 71
> > 	Region 0: Memory at 000000009ffff000 (32-bit, non-prefetchable)
> > [size=4K]
> > 	Region 1: Memory at 000000009fc00000 (32-bit, non-prefetchable)
> > [size=2M]
> >
> > But when I try to access resource0 or resource1, I get a read error.
> > What characteristic of the device or driver determines whether it will
> > allow mmap-ing?
> >
> > (I've written the driver for this device myself.)
> 
> Nothing special beyond normal unix perms on the resource[0..n] files  
> to my knowledge.  When you say you get a read error what exactly does  
> that mean?

It means I had a bug in my program which read (mmap()ed) the resouce :-(
It is fixed now (see below).  Thanks for the tip.

-- 
Phil Nitschke <Phil.Nitschke@avalon.com.au>
Avalon Systems Pty Ltd



[-- Attachment #2: mmap.c --]
[-- Type: text/x-csrc, Size: 1009 bytes --]

#include <assert.h>
#include <byteswap.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <sys/mman.h>
#include <unistd.h>

int
main (int argc, char **argv)
{
  /* Define constants for specific to the Base Address Register(s)
   * (defining the PCI I/O Address Region) for our hardware. */
  const uint32_t BAR_LENGTH = 0x200000;
  const uint32_t CTRL_REGS  =  0x80000;
  const uint32_t CTRL_LEN   =     0x30;

  int i, fd;
  uint32_t *ptr;

  fd = open("/sys/bus/pci/devices/0000:00:01.0/resource1", O_RDWR);
  ptr = (uint32_t*) mmap(NULL, BAR_LENGTH, PROT_READ|PROT_WRITE,
			 MAP_SHARED, fd, 0);
  // If ptr == -1, then the user was probably not root
  assert(ptr != MAP_FAILED);

  printf ("Mapped %p bytes beginning at address %p\n", BAR_LENGTH, ptr);

  for (i = CTRL_REGS; i < (CTRL_REGS + CTRL_LEN); i += 4)
    printf ("mmap[0x%08x] (%p) = 0x%08x [0x%08x]\n",
	    i, ptr + i/4, *(ptr + i/4), bswap_32(*(ptr + i/4)));

  munmap(ptr, BAR_LENGTH);
  close(fd);
}

Re: Memory mapping PCI memory region to user space

[David Hawkins]

> I'm trying to map a PCI memory region 1 into user space from my driver 
> (PPC440GX, Linux 2.6.10).  Here's the mmap routine of the driver that 
> I'm using:

Hi Chris,

I wrote a generic PCI IO driver when testing the
Yosemite 440EP, take a look at the code, its fairly
well commented. You probably just need to add the
PCI vendor ID for your target board:
(or you can echo the deviceID:vendorID to the
sysfs node for the driver)

http://www.ovro.caltech.edu/~dwh/pci_io.tar.gz

There's some comments in here on it:
http://www.ovro.caltech.edu/~dwh/correlator/pdf/LNX-723-Hawkins.pdf

For example, on an x86 system, I was looking for
a PLX PCI9054 10b5:9054, and then on the yosemite
board I was looking for the same hardware.

In your case, its not clear from your email whether you
are on a 440GX looking for another device, or on a host
looking for a 440GX. Either way, the driver should work.

Cheers
Dave

RE: Memory mapping PCI memory region to user space

[Wyse, Chris]

Hi,

Thanks for all the help.

My ideal solution would have been to use sysfs, but I didn't want to
upgrade the kernel.  Therefore, I used Dave's program as a guide for
setting the protect bits appropriately, and was able to get the driver
working.  The driver required the pfn_pte macro change specified in my
original post.  It appears to be a kernel bug on PPC processors with a
greater than 32 bit address space.


Chris Wyse
Member of Technical Staff
Embedded Technologies
860-749-1556 office
860-978-0849 cell
413-778-9101 fax
http://www.windriver.com
 =20


-----Original Message-----
From: David Hawkins [mailto:dwh@ovro.caltech.edu]=20
Sent: Thursday, March 23, 2006 12:47 PM
To: Kumar Gala
Cc: Wyse, Chris; Linuxppc-Embedded ((E-Mail))
Subject: Re: Memory mapping PCI memory region to user space

Hi Kumar,

>> When I was testing the Yosemite board as the host, I found that I=20
>> could set the endian flag on the mmapped page, which then made the=20
>> PCI device registers read as 32-bit quantities read back with the=20
>> same layout under both x86 and PPC hosts.
>=20
> Hmm, I guess I would handle this like how the reset of the kernel=20
> handle is with the io routines handling the swapping.  Not sure if=20
> there is any advantage to using the endian flag.  I guess if you have=20
> something you are treating as just memory there would be.

I haven't used the feature, I just tested it to see what it did.

The application case I thought of was this; the PCI boards I built (that
I am revising, and replacing the DSP with a PPC) have an 8MB PCI region
that I can mmap from the host. I have a test suite that runs from the
host that manipulates registers on the boards to download FPGAs etc.
When the boards are used in a real system, the onboard DSP is generally
used, and the host just talks to the DSP.

However, for the test suite, if I have a header with definitions
like:

#define CONTROL_FPGA_ENABLE   (1 << 0)
#define CONTROL_FPGA_DONE_BIT (1 << 1)

that correspond to bits in a 32-bit PCI mmapped register. Then code in
the user-space test suite that did something like

   pci_addr[CONTROL_OFFSET] |=3D CONTROL_FPGA_ENABLE;

would instead need to be re-written, eg.,

   write_le32(&pci_addr[CONTROL_OFFSET], CONTROL_FPGA_ENABLE);

to be portable.

I definitely agree that this is how kernel-level code should be written,
but user-space code ... well, if I want to reuse code already written,
setting the page endian flag and reusing the code would seem like the
way to go. (This isn't what I need to do, since my host will still be an
x86, the PPC will be a target device, but I still need to think about
the endian issues).

Now of course that I have seen the consequences of my coding, I'll be
more careful to deal with endianness more appropriately.

Its a tricky trade-off though. I could define control ioctl's that hide
all the endianness issues ... but then the driver just gets bigger. I
think the appropriate solution for the user-space test code would be to
use CPU-to-little-endian routines, and wrap the lot in a re-usable
library that the test suite links against.

> There isn't a sysfs flag for the endianness page attribute since =20
> thats a PPC book-e specific feature.  We could possible expand things

> to support it but, I've been trying to actively avoid using the 'E'
bit.

Ok, I haven't received the 8349E board that I am waiting on, so I hadn't
spotted that the PAGE_ENDIAN flag was Book E specific.

Thanks for your insight.

Cheers
Dave

Re: Memory mapping PCI memory region to user space

[Kumar Gala]

On Mar 23, 2006, at 1:52 PM, Wyse, Chris wrote:

> Hi,
>
> Thanks for all the help.
>
> My ideal solution would have been to use sysfs, but I didn't want to
> upgrade the kernel.  Therefore, I used Dave's program as a guide for
> setting the protect bits appropriately, and was able to get the driver
> working.  The driver required the pfn_pte macro change specified in my
> original post.  It appears to be a kernel bug on PPC processors with a
> greater than 32 bit address space.

The issue with greater than 32 bit address spaces goes away in newer  
kernels with the use of io_remap_pfn_range() instead of  
io_remap_page_range().

- kumar

> -----Original Message-----
> From: David Hawkins [mailto:dwh@ovro.caltech.edu]
> Sent: Thursday, March 23, 2006 12:47 PM
> To: Kumar Gala
> Cc: Wyse, Chris; Linuxppc-Embedded ((E-Mail))
> Subject: Re: Memory mapping PCI memory region to user space
>
> Hi Kumar,
>
>>> When I was testing the Yosemite board as the host, I found that I
>>> could set the endian flag on the mmapped page, which then made the
>>> PCI device registers read as 32-bit quantities read back with the
>>> same layout under both x86 and PPC hosts.
>>
>> Hmm, I guess I would handle this like how the reset of the kernel
>> handle is with the io routines handling the swapping.  Not sure if
>> there is any advantage to using the endian flag.  I guess if you have
>> something you are treating as just memory there would be.
>
> I haven't used the feature, I just tested it to see what it did.
>
> The application case I thought of was this; the PCI boards I built  
> (that
> I am revising, and replacing the DSP with a PPC) have an 8MB PCI  
> region
> that I can mmap from the host. I have a test suite that runs from the
> host that manipulates registers on the boards to download FPGAs etc.
> When the boards are used in a real system, the onboard DSP is  
> generally
> used, and the host just talks to the DSP.
>
> However, for the test suite, if I have a header with definitions
> like:
>
> #define CONTROL_FPGA_ENABLE   (1 << 0)
> #define CONTROL_FPGA_DONE_BIT (1 << 1)
>
> that correspond to bits in a 32-bit PCI mmapped register. Then code in
> the user-space test suite that did something like
>
>    pci_addr[CONTROL_OFFSET] |= CONTROL_FPGA_ENABLE;
>
> would instead need to be re-written, eg.,
>
>    write_le32(&pci_addr[CONTROL_OFFSET], CONTROL_FPGA_ENABLE);
>
> to be portable.
>
> I definitely agree that this is how kernel-level code should be  
> written,
> but user-space code ... well, if I want to reuse code already written,
> setting the page endian flag and reusing the code would seem like the
> way to go. (This isn't what I need to do, since my host will still  
> be an
> x86, the PPC will be a target device, but I still need to think about
> the endian issues).
>
> Now of course that I have seen the consequences of my coding, I'll be
> more careful to deal with endianness more appropriately.
>
> Its a tricky trade-off though. I could define control ioctl's that  
> hide
> all the endianness issues ... but then the driver just gets bigger. I
> think the appropriate solution for the user-space test code would  
> be to
> use CPU-to-little-endian routines, and wrap the lot in a re-usable
> library that the test suite links against.
>
>> There isn't a sysfs flag for the endianness page attribute since
>> thats a PPC book-e specific feature.  We could possible expand things
>
>> to support it but, I've been trying to actively avoid using the 'E'
> bit.
>
> Ok, I haven't received the 8349E board that I am waiting on, so I  
> hadn't
> spotted that the PAGE_ENDIAN flag was Book E specific.
>
> Thanks for your insight.
>
> Cheers
> Dave
>
>
>