Create permanent mapping from PCI bus to region of physical memory

Create permanent mapping from PCI bus to region of physical memory

[Howard, Marc]

Hi,

I'm working on a PPC440GX based board in which a PCI based peripheral
communicates with the host via a shared region of processor memory.  The
peripheral will read and write this region autonomously.  Because of
this I need a fixed mapping FROM the PCI bus TO system memory.

It's easy enough to rope off the top 16 MB or so of system physical
memory by lying to linux in the boot arguments.  What I can't seem find
is a system call to use to create a fixed, permanent mapping of PPC
memory as seen from the PCI bus.

Has anyone out there done this and could share a code snippet with me?

Thanks,

Marc W. Howard

Re: Create permanent mapping from PCI bus to region of physical memory

[David Hawkins]

Howard, Marc wrote:
> Hi,
> 
> I'm working on a PPC440GX based board in which a PCI based peripheral
> communicates with the host via a shared region of processor memory.  The
> peripheral will read and write this region autonomously.  Because of
> this I need a fixed mapping FROM the PCI bus TO system memory.
> 
> It's easy enough to rope off the top 16 MB or so of system physical
> memory by lying to linux in the boot arguments.  What I can't seem find
> is a system call to use to create a fixed, permanent mapping of PPC
> memory as seen from the PCI bus.
> 
> Has anyone out there done this and could share a code snippet with me?
> 

Hi Marc,

So the PPC440GX is the host, but what is the peripheral, and
what restricts it to require a fixed address? Can it handle
a set of fixed addresses, eg. a scatter-gather buffer setup
by the host? Can the host and peripheral communicate in any
other way, eg. when the peripheral changes something in the
shared memory, surely it has to interrupt the host to let it
know?

Does the peripheral contain a CPU, if thats the case, then
the host and peripheral could also maintain a smaller region
that contains addresses of other pages, i.e., no need to
restrict the design to a single block of memory, you just
need a page of pointers to other shared blocks of pages.

For example, the alloc_pages call can be used to get order 11
pages, eg. 2^11 x 4096-byte pages, 8MB (hmm, thats sounds a bit
big, I seem to recall 2MB was the maximum). So your 16MBs
could be 8 separate blocks.

Anyway the comment is that you can use alloc_pages() to give
you a contiguous chunk of memory at some allocation-time
address, but not a 16MB chunk.

Of course if the peripheral is fairly dumb, then you can't do
this. But if it indicates a buffer-used condition, perhaps you
can swap over to a new 2MB block.

If you can provide a few more details on the restriction of
the peripheral, then I or others can suggest options.

Dave

RE: Create permanent mapping from PCI bus to region of physical memory

[Howard, Marc]

[-- Attachment #1: Type: text/plain, Size: 2082 bytes --]

Howard, Marc wrote:
>> Hi,
>> 
>> I'm working on a PPC440GX based board in which a PCI based peripheral
>> communicates with the host via a shared region of processor memory.  The
>> peripheral will read and write this region autonomously.  Because of
>> this I need a fixed mapping FROM the PCI bus TO system memory.
>>
>> It's easy enough to rope off the top 16 MB or so of system physical
>> memory by lying to linux in the boot arguments.  What I can't seem find
>> is a system call to use to create a fixed, permanent mapping of PPC
>> memory as seen from the PCI bus.
>> 
>> Has anyone out there done this and could share a code snippet with me?
>> 

>Hi Marc,

>So the PPC440GX is the host, but what is the peripheral, and
>what restricts it to require a fixed address? Can it handle
>a set of fixed addresses, eg. a scatter-gather buffer setup
>by the host? Can the host and peripheral communicate in any
>other way, eg. when the peripheral changes something in the
>shared memory, surely it has to interrupt the host to let it
>know?

>Does the peripheral contain a CPU, if thats the case, then
>the host and peripheral could also maintain a smaller region
>that contains addresses of other pages, i.e., no need to
>restrict the design to a single block of memory, you just
>need a page of pointers to other shared blocks of pages.

Dave,

The periperal is an FPGA.  No, there is no internal processor;
everything is coded in Verilog.

Scatter/gather isn't a viable option because of this.  Additionally
non-contiguous memory would reduce bandwidth and increase
FPGA design complexity.  The data must be contignuous because
of these reasons and the need for the data to be randomly
accessible from the outside using simple address arithmetic.

I realize this isn't a standard linux request but having
fixed, linear memory is quite common in embedded apps.  There
should be a way to create this mapping in the 440GX's hardware
and I'm just looking for a system call (if there is one) to
implement it.

Thanks,

Marc




[-- Attachment #2: Type: text/html, Size: 2869 bytes --]

Re: Create permanent mapping from PCI bus to region of physical memory

[David Hawkins]

Hi Marc,

> The periperal is an FPGA.  No, there is no internal processor;
> everything is coded in Verilog.
>
> Scatter/gather isn't a viable option because of this. 

Er, why not, its an FPGA, everything is possible :)

So you have a PCI core, since you are planning to write to
the host memory space, its a Bus Master PCI core.
Who's FPGA, Altera, Xilinx, or someone else?

> Additionally non-contiguous memory would reduce bandwidth 
> and increase FPGA design complexity. 

Not necessarily. If the target is using bus master DMA to
write to the host memory, then you can hit pretty close
to the bandwidth of the PCI bus. If you are DMAing in
big blocks, the overhead of a block change isn't too much.
I did tests with the 440EP using a DMA controller on an
adapter board and found that the PCI bridge in the 440EP
was the limiting factor, i.e., for a 33MHz 32-bit bus
with a potential for 132MB/s, the *best* you can do is
about 40MB/s since the bridge only accepts data in cache
line sizes before sending a retry to the target. I can
send you those results.

> The data must be contignuous because
> of these reasons and the need for the data to be randomly
> accessible from the outside using simple address arithmetic.

Randomly accessible from where; the host or an I/O interface
at the FPGA. The pages can be made to appear contiguous to
a host processor user-space process using the nopage callback
of the VMA.

> I realize this isn't a standard linux request but having
> fixed, linear memory is quite common in embedded apps.  There
> should be a way to create this mapping in the 440GX's hardware
> and I'm just looking for a system call (if there is one) to
> implement it.

Alas, this is one of the concessions one must make if you
want to use a processor that enables the MMU. However,
I don't see any fundamental limitation in the design
that would preclude a little extra work on the FPGA.
But, it does require additional Verilog to support
the flexibility. The long-term advantage is that you
don't have to provide a hack (eg. reserve a block of
high-memory under Linux).

So how about this concession. If Linux lets you alloc_pages
in 2MB max chunks, create 8 address decode regions on your
FPGA. Provide the host access to the 8 address registers.
When the Linux driver is installed, the driver alloc_pages
8 times and loads the PCI address of those regions into
the 8 registers.

On the FPGA side of things create a flat 16MB region, as
an address passes through a 2MB block, it changes the PCI
address it decodes to. So, your FPGA believes it has
a 16MB continuous block, and Linux can supply the memory
as 8 non-contiguous chunks. Back in user-space on Linux,
the nopage VMA call is used to map a page-at-a-time
and the 2MB regions appear as a 16MB contiguous region to
user-space. (There are probably other ways to make user
space see it as one block too)

Cheers
Dave

RE: Create permanent mapping from PCI bus to region of physical memory

[Howard, Marc]

[-- Attachment #1: Type: text/plain, Size: 3052 bytes --]

>> The periperal is an FPGA.  No, there is no internal processor;
>> everything is coded in Verilog.
>>
>> Scatter/gather isn't a viable option because of this. 

>Er, why not, its an FPGA, everything is possible :)

Not if you're tight on routing resouces and/or design time ;-)

>So you have a PCI core, since you are planning to write to
>the host memory space, its a Bus Master PCI core.
>Who's FPGA, Altera, Xilinx, or someone else?

Xilinx Virtex 4, no internal PPC.  The vendor really isn't
germain to the problem at hand however.

>> Additionally non-contiguous memory would reduce bandwidth 
>> and increase FPGA design complexity. 

>Not necessarily. If the target is using bus master DMA to
>write to the host memory, then you can hit pretty close
>to the bandwidth of the PCI bus. If you are DMAing in
>big blocks, the overhead of a block change isn't too much.
>I did tests with the 440EP using a DMA controller on an
>adapter board and found that the PCI bridge in the 440EP
>was the limiting factor, i.e., for a 33MHz 32-bit bus
>with a potential for 132MB/s, the *best* you can do is
>about 40MB/s since the bridge only accepts data in cache
>line sizes before sending a retry to the target. I can
>send you those results.

Dave, I appreciate your input but scatter/gather just isn't
an option here for a variety of reasons.  Bandwidth/complexity/
latency/time to design/time to debug/FPGA density all
factor into this decision.  BTW we're talking 66/64 PCI-X;
33/32 PCI isn't nearly fast enough.

>Randomly accessible from where; the host or an I/O interface
>at the FPGA. The pages can be made to appear contiguous to
>a host processor user-space process using the nopage callback
>of the VMA.

>From the point of view of the FPGA.

>> I realize this isn't a standard linux request but having
>> fixed, linear memory is quite common in embedded apps.  There
>> should be a way to create this mapping in the 440GX's hardware
>> and I'm just looking for a system call (if there is one) to
>> implement it.

>Alas, this is one of the concessions one must make if you
>want to use a processor that enables the MMU. 

Nope, I disagree.  The MMU isn't involved here at all.  I'm
talking about setting up the PIM (PCI Inbound Map) via
a system call as opposed to just writing it directly.

>However,
>I don't see any fundamental limitation in the design
>that would preclude a little extra work on the FPGA.
>But, it does require additional Verilog to support
>the flexibility. The long-term advantage is that you
>don't have to provide a hack (eg. reserve a block of
>high-memory under Linux).

I really can't go into detail about my application for a
variety of reasons (both technical and legal) but suffice
it to say that 16 MB is just the starting point.

I guess I'll dig a little deeper into the source to see
where the kernel does the PIM mapping.  Re-architecting
our app at this stage just isn't a practical consideration.

Thanks again,

Marc 






[-- Attachment #2: Type: text/html, Size: 3973 bytes --]

Re: Create permanent mapping from PCI bus to region of physical memory

[David Hawkins]

> Not if you're tight on routing resouces and/or design time ;-)

Yep, understood.

> Dave, I appreciate your input but scatter/gather just isn't
> an option here for a variety of reasons.  Bandwidth/complexity/
> latency/time to design/time to debug/FPGA density all
> factor into this decision.  BTW we're talking 66/64 PCI-X;
> 33/32 PCI isn't nearly fast enough.

No problem.

> Nope, I disagree.  The MMU isn't involved here at all.  I'm
> talking about setting up the PIM (PCI Inbound Map) via
> a system call as opposed to just writing it directly.

The MMU is involved since its responsible for all memory,
i.e., Linux is told that all physical memory is under
its control. I haven't looked at the PCI inbound map,
but I would assume that it can be setup to say map all
of the incoming PCI addresses to all of the SDRAM
addresses, or some block thereof. So the FPGA can see physical
pages that Linux is busy swapping data into and out of.
What you are wanting to do is tell Linux to lay-off
managing a block of SDRAM, but make it visible to the
kernel as a memory area.

So, you need the kernel to setup page tables that map
to those physical addresses and you need to tell Linux
not to treat them like it does the rest of SDRAM.

So I believe the MMU/page tables are involved.

> I really can't go into detail about my application for a
> variety of reasons (both technical and legal) but suffice
> it to say that 16 MB is just the starting point.

No sweat, it never hurts to ask.

> I guess I'll dig a little deeper into the source to see
> where the kernel does the PIM mapping.  Re-architecting
> our app at this stage just isn't a practical consideration.

You always have the hack of reserving a block of memory
at the top of SDRAM, i.e., lie to Linux and tell it it
has less memory. It sounds like that might be the least
effort. Rubini has an example of how to do that.

Dave

Re: Create permanent mapping from PCI bus to region of physical memory

[Mark Chambers]

> You always have the hack of reserving a block of memory
> at the top of SDRAM, i.e., lie to Linux and tell it it
> has less memory. It sounds like that might be the least
> effort. Rubini has an example of how to do that.
> 

I've done this with a bus-master PCI peripheral and it works
nicely.  The other point about this is that the mmap() call
will work rationally with memory above the linux pool, so
it's easy to go straight to/from user land with this approach.
(and you can set caching and the guard bit and all that so
that PPC access works the way you want it to)

Mark Chambers